good_job 1.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +153 -0
- data/LICENSE.txt +20 -0
- data/README.md +242 -0
- data/exe/good_job +3 -0
- data/lib/active_job/queue_adapters/good_job_adapter.rb +19 -0
- data/lib/good_job.rb +19 -0
- data/lib/good_job/adapter.rb +57 -0
- data/lib/good_job/cli.rb +91 -0
- data/lib/good_job/job.rb +106 -0
- data/lib/good_job/lockable.rb +106 -0
- data/lib/good_job/logging.rb +70 -0
- data/lib/good_job/performer.rb +12 -0
- data/lib/good_job/pg_locks.rb +21 -0
- data/lib/good_job/railtie.rb +7 -0
- data/lib/good_job/scheduler.rb +79 -0
- data/lib/good_job/version.rb +3 -0
- metadata +285 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 9e068306c80c1080f736520219cdf27230f9684bd69fa70cff0489ce37b1ed79
|
4
|
+
data.tar.gz: 8f10fb768248ca1ec1e14f44be2073f9c60069b2451c5858c571efbca84c0969
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 8ebe33709d71b2254fa2fc0dbc1c991df739a67ba6ee3649000bada6c4618b2d958c58372818502b57211fb4694b28b16be513227220b262da2125b6f87b61bb
|
7
|
+
data.tar.gz: 40db6accac1448cef4cfff7516e68d401b25711520d479cf3af4c20f03a31526f2e084aed5dabc64adceb9b23ae75ac41ae74e25bb80a4bb80c320a31fc9fb25
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,153 @@
|
|
1
|
+
# Changelog
|
2
|
+
|
3
|
+
## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-21)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.0...v1.0.1)
|
6
|
+
|
7
|
+
**Merged pull requests:**
|
8
|
+
|
9
|
+
- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
|
10
|
+
|
11
|
+
## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
|
12
|
+
|
13
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.9.0...v1.0.0)
|
14
|
+
|
15
|
+
## [v0.9.0](https://github.com/bensheldon/good_job/tree/v0.9.0) (2020-07-20)
|
16
|
+
|
17
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
|
18
|
+
|
19
|
+
**Merged pull requests:**
|
20
|
+
|
21
|
+
- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
|
22
|
+
|
23
|
+
## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
|
24
|
+
|
25
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)
|
26
|
+
|
27
|
+
**Closed issues:**
|
28
|
+
|
29
|
+
- Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
|
30
|
+
|
31
|
+
**Merged pull requests:**
|
32
|
+
|
33
|
+
- Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
|
34
|
+
- Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
|
35
|
+
- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
|
36
|
+
|
37
|
+
## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-18)
|
38
|
+
|
39
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.0...v0.8.1)
|
40
|
+
|
41
|
+
## [v0.8.0](https://github.com/bensheldon/good_job/tree/v0.8.0) (2020-07-17)
|
42
|
+
|
43
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.7.0...v0.8.0)
|
44
|
+
|
45
|
+
**Merged pull requests:**
|
46
|
+
|
47
|
+
- Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
|
48
|
+
|
49
|
+
## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)
|
50
|
+
|
51
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
|
52
|
+
|
53
|
+
**Closed issues:**
|
54
|
+
|
55
|
+
- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
|
56
|
+
|
57
|
+
**Merged pull requests:**
|
58
|
+
|
59
|
+
- Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
|
60
|
+
- Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
|
61
|
+
- Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
|
62
|
+
|
63
|
+
## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
|
64
|
+
|
65
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.5.0...v0.6.0)
|
66
|
+
|
67
|
+
**Closed issues:**
|
68
|
+
|
69
|
+
- Improve the command line options [\#32](https://github.com/bensheldon/good_job/issues/32)
|
70
|
+
- Allow config.active\_job.queue\_adapter = :good\_job to work [\#5](https://github.com/bensheldon/good_job/issues/5)
|
71
|
+
|
72
|
+
**Merged pull requests:**
|
73
|
+
|
74
|
+
- Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
|
75
|
+
- Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
|
76
|
+
- Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
|
77
|
+
- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
|
78
|
+
- Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
|
79
|
+
|
80
|
+
## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
|
81
|
+
|
82
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.4.0...v0.5.0)
|
83
|
+
|
84
|
+
**Merged pull requests:**
|
85
|
+
|
86
|
+
- Update development Ruby to 2.6.6 and gems [\#29](https://github.com/bensheldon/good_job/pull/29) ([bensheldon](https://github.com/bensheldon))
|
87
|
+
|
88
|
+
## [v0.4.0](https://github.com/bensheldon/good_job/tree/v0.4.0) (2020-03-31)
|
89
|
+
|
90
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.3.0...v0.4.0)
|
91
|
+
|
92
|
+
**Merged pull requests:**
|
93
|
+
|
94
|
+
- Improve ActiveRecord usage for advisory locking [\#24](https://github.com/bensheldon/good_job/pull/24) ([bensheldon](https://github.com/bensheldon))
|
95
|
+
- Remove support for Rails 5.1 [\#23](https://github.com/bensheldon/good_job/pull/23) ([bensheldon](https://github.com/bensheldon))
|
96
|
+
|
97
|
+
## [v0.3.0](https://github.com/bensheldon/good_job/tree/v0.3.0) (2020-03-22)
|
98
|
+
|
99
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.2...v0.3.0)
|
100
|
+
|
101
|
+
**Merged pull requests:**
|
102
|
+
|
103
|
+
- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
|
104
|
+
- Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
|
105
|
+
- Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
|
106
|
+
- Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
|
107
|
+
|
108
|
+
## [v0.2.2](https://github.com/bensheldon/good_job/tree/v0.2.2) (2020-03-08)
|
109
|
+
|
110
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...v0.2.2)
|
111
|
+
|
112
|
+
**Merged pull requests:**
|
113
|
+
|
114
|
+
- Gracefully shutdown Scheduler when executable receives TERM or INT [\#17](https://github.com/bensheldon/good_job/pull/17) ([bensheldon](https://github.com/bensheldon))
|
115
|
+
- Update Appraisals [\#16](https://github.com/bensheldon/good_job/pull/16) ([bensheldon](https://github.com/bensheldon))
|
116
|
+
|
117
|
+
## [v0.2.1](https://github.com/bensheldon/good_job/tree/v0.2.1) (2020-03-07)
|
118
|
+
|
119
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.0...v0.2.1)
|
120
|
+
|
121
|
+
**Merged pull requests:**
|
122
|
+
|
123
|
+
- Clean up Gemspec [\#15](https://github.com/bensheldon/good_job/pull/15) ([bensheldon](https://github.com/bensheldon))
|
124
|
+
- Set up Rubocop [\#14](https://github.com/bensheldon/good_job/pull/14) ([bensheldon](https://github.com/bensheldon))
|
125
|
+
- Add pg gem as explicit dependency [\#13](https://github.com/bensheldon/good_job/pull/13) ([bensheldon](https://github.com/bensheldon))
|
126
|
+
- Bump nokogiri from 1.10.7 to 1.10.9 [\#12](https://github.com/bensheldon/good_job/pull/12) ([dependabot[bot]](https://github.com/apps/dependabot))
|
127
|
+
- Add Appraisal with tests for Rails 5.1, 5.2, 6.0 [\#11](https://github.com/bensheldon/good_job/pull/11) ([bensheldon](https://github.com/bensheldon))
|
128
|
+
|
129
|
+
## [v0.2.0](https://github.com/bensheldon/good_job/tree/v0.2.0) (2020-03-06)
|
130
|
+
|
131
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.1.0...v0.2.0)
|
132
|
+
|
133
|
+
**Merged pull requests:**
|
134
|
+
|
135
|
+
- Use Rails.logger and ActiveSupport::Notifications for logging instead of puts [\#10](https://github.com/bensheldon/good_job/pull/10) ([bensheldon](https://github.com/bensheldon))
|
136
|
+
- Remove minitest files [\#9](https://github.com/bensheldon/good_job/pull/9) ([bensheldon](https://github.com/bensheldon))
|
137
|
+
- Use scheduled\_at and priority for scheduling [\#8](https://github.com/bensheldon/good_job/pull/8) ([bensheldon](https://github.com/bensheldon))
|
138
|
+
- Create Github Action workflow for PRs and Issues [\#7](https://github.com/bensheldon/good_job/pull/7) ([bensheldon](https://github.com/bensheldon))
|
139
|
+
|
140
|
+
## [v0.1.0](https://github.com/bensheldon/good_job/tree/v0.1.0) (2020-03-03)
|
141
|
+
|
142
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/6866006239f1a6b7fcb7103f5df60d904952fb84...v0.1.0)
|
143
|
+
|
144
|
+
**Merged pull requests:**
|
145
|
+
|
146
|
+
- Add executable with Thor [\#4](https://github.com/bensheldon/good_job/pull/4) ([bensheldon](https://github.com/bensheldon))
|
147
|
+
- Refactor adapter enqueing methods; expand Readme, tests, editorconfig [\#3](https://github.com/bensheldon/good_job/pull/3) ([bensheldon](https://github.com/bensheldon))
|
148
|
+
- Fetch new jobs within the worker thread itself; incrementally grow worker threads [\#2](https://github.com/bensheldon/good_job/pull/2) ([bensheldon](https://github.com/bensheldon))
|
149
|
+
- Set up Github Workflows for tests [\#1](https://github.com/bensheldon/good_job/pull/1) ([bensheldon](https://github.com/bensheldon))
|
150
|
+
|
151
|
+
|
152
|
+
|
153
|
+
\* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
|
data/LICENSE.txt
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
Copyright 2020 Ben Sheldon
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
4
|
+
a copy of this software and associated documentation files (the
|
5
|
+
"Software"), to deal in the Software without restriction, including
|
6
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
7
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
8
|
+
permit persons to whom the Software is furnished to do so, subject to
|
9
|
+
the following conditions:
|
10
|
+
|
11
|
+
The above copyright notice and this permission notice shall be
|
12
|
+
included in all copies or substantial portions of the Software.
|
13
|
+
|
14
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
15
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
16
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
17
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
18
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
19
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
20
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,242 @@
|
|
1
|
+
# GoodJob
|
2
|
+
|
3
|
+
GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
|
4
|
+
|
5
|
+
**Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.**
|
6
|
+
|
7
|
+
- **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration.
|
8
|
+
- **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
|
9
|
+
- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
|
10
|
+
- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
|
11
|
+
|
12
|
+
## Installation
|
13
|
+
|
14
|
+
Add this line to your application's Gemfile:
|
15
|
+
|
16
|
+
```ruby
|
17
|
+
gem 'good_job'
|
18
|
+
```
|
19
|
+
|
20
|
+
And then execute:
|
21
|
+
```bash
|
22
|
+
$ bundle install
|
23
|
+
```
|
24
|
+
|
25
|
+
## Usage
|
26
|
+
|
27
|
+
1. Create a database migration:
|
28
|
+
```bash
|
29
|
+
$ bin/rails g migration CreateGoodJobs
|
30
|
+
```
|
31
|
+
|
32
|
+
Add to the newly created migration file:
|
33
|
+
|
34
|
+
```ruby
|
35
|
+
class CreateGoodJobs < ActiveRecord::Migration[6.0]
|
36
|
+
def change
|
37
|
+
enable_extension 'pgcrypto'
|
38
|
+
|
39
|
+
create_table :good_jobs, id: :uuid do |t|
|
40
|
+
t.timestamps
|
41
|
+
|
42
|
+
t.text :queue_name
|
43
|
+
t.integer :priority
|
44
|
+
t.jsonb :serialized_params
|
45
|
+
t.timestamp :scheduled_at
|
46
|
+
t.timestamp :performed_at
|
47
|
+
t.timestamp :finished_at
|
48
|
+
t.text :error
|
49
|
+
|
50
|
+
add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
|
51
|
+
add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
```
|
56
|
+
|
57
|
+
Run the migration:
|
58
|
+
|
59
|
+
```bash
|
60
|
+
$ bin/rails db:migrate
|
61
|
+
```
|
62
|
+
|
63
|
+
1. Configure the ActiveJob adapter:
|
64
|
+
```ruby
|
65
|
+
# config/application.rb
|
66
|
+
config.active_job.queue_adapter = :good_job
|
67
|
+
```
|
68
|
+
|
69
|
+
By default, using `:good_job` is equivalent to manually configuring the adapter:
|
70
|
+
|
71
|
+
```ruby
|
72
|
+
# config/environments/development.rb
|
73
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
74
|
+
|
75
|
+
# config/environments/test.rb
|
76
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
|
77
|
+
|
78
|
+
# config/environments/production.rb
|
79
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
|
80
|
+
```
|
81
|
+
|
82
|
+
1. Queue your job 🎉:
|
83
|
+
```ruby
|
84
|
+
YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
|
85
|
+
```
|
86
|
+
|
87
|
+
1. In production, the scheduler is designed to run in its own process:
|
88
|
+
```bash
|
89
|
+
$ bundle exec good_job
|
90
|
+
```
|
91
|
+
|
92
|
+
Configuration options available with `help`:
|
93
|
+
```bash
|
94
|
+
$ bundle exec good_job help start
|
95
|
+
|
96
|
+
# Usage:
|
97
|
+
# good_job start
|
98
|
+
#
|
99
|
+
# Options:
|
100
|
+
# [--max-threads=N] # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
|
101
|
+
# [--queues=queue1,queue2] # Queues to work from. Separate multiple queues with commas (default: *)
|
102
|
+
# [--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
|
103
|
+
```
|
104
|
+
|
105
|
+
### Taking advantage of ActiveJob
|
106
|
+
|
107
|
+
ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
|
108
|
+
|
109
|
+
```ruby
|
110
|
+
class ApplicationJob < ActiveJob::Base
|
111
|
+
# Retry errors an infinite number of times with exponential back-off
|
112
|
+
retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
|
113
|
+
|
114
|
+
# Timeout jobs after 10 minutes
|
115
|
+
JobTimeoutError = Class.new(StandardError)
|
116
|
+
around_perform do |_job, block|
|
117
|
+
Timeout.timeout(10.minutes, JobTimeoutError) do
|
118
|
+
block.call
|
119
|
+
end
|
120
|
+
end
|
121
|
+
end
|
122
|
+
```
|
123
|
+
|
124
|
+
### Configuring Job Execution Threads
|
125
|
+
|
126
|
+
GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
|
127
|
+
|
128
|
+
- Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._
|
129
|
+
- The maximum number of GoodJob threads can be configured, in decreasing precedence:
|
130
|
+
1. `$ bundle exec good_job --max_threads 4`
|
131
|
+
2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
|
132
|
+
3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
|
133
|
+
4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
|
134
|
+
|
135
|
+
### Migrating to GoodJob from a different ActiveJob backend
|
136
|
+
|
137
|
+
If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
|
138
|
+
|
139
|
+
1. Enqueue newly created jobs on GoodJob either entirely by setting `ActiveJob::Base.queue_adapter = :good_job` or progressively via individual job classes:
|
140
|
+
|
141
|
+
```ruby
|
142
|
+
# jobs/specific_job.rb
|
143
|
+
class SpecificJob < ApplicationJob
|
144
|
+
self.queue_adapter = :good_job
|
145
|
+
# ...
|
146
|
+
end
|
147
|
+
```
|
148
|
+
|
149
|
+
1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
|
150
|
+
```procfile
|
151
|
+
# Procfile
|
152
|
+
# ...
|
153
|
+
worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
|
154
|
+
```
|
155
|
+
|
156
|
+
1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
|
157
|
+
|
158
|
+
### Monitoring and preserving worked jobs
|
159
|
+
|
160
|
+
GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
|
161
|
+
|
162
|
+
By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`).
|
163
|
+
|
164
|
+
To preserve job records for later inspection, set an initializer:
|
165
|
+
|
166
|
+
```ruby
|
167
|
+
# config/initializers/good_job.rb
|
168
|
+
GoodJob.preserve_job_records = true
|
169
|
+
```
|
170
|
+
|
171
|
+
It is also necessary to delete these preserved jobs from the database after a certain time period:
|
172
|
+
|
173
|
+
- For example, in a Rake task:
|
174
|
+
|
175
|
+
```ruby
|
176
|
+
# GoodJob::Job.finished(1.day.ago).delete_all
|
177
|
+
```
|
178
|
+
- For example, using the `good_job` command-line utility:
|
179
|
+
|
180
|
+
```bash
|
181
|
+
$ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
|
182
|
+
```
|
183
|
+
|
184
|
+
## Development
|
185
|
+
|
186
|
+
To run tests:
|
187
|
+
|
188
|
+
```bash
|
189
|
+
# Clone the repository locally
|
190
|
+
$ git clone git@github.com:bensheldon/good_job.git
|
191
|
+
|
192
|
+
# Set up the local environment
|
193
|
+
$ bin/setup_test
|
194
|
+
|
195
|
+
# Run the tests
|
196
|
+
$ bin/rspec
|
197
|
+
```
|
198
|
+
|
199
|
+
This gem uses Appraisal to run tests against multiple versions of Rails:
|
200
|
+
|
201
|
+
```bash
|
202
|
+
# Install Appraisal(s) gemfiles
|
203
|
+
$ bundle exec appraisal
|
204
|
+
|
205
|
+
# Run tests
|
206
|
+
$ bundle exec appraisal bin/rspec
|
207
|
+
|
208
|
+
```
|
209
|
+
|
210
|
+
For developing locally within another Ruby on Rails project:
|
211
|
+
|
212
|
+
```bash
|
213
|
+
# Within Ruby on Rails directory...
|
214
|
+
$ bundle config local.good_job /path/to/local/git/repository
|
215
|
+
|
216
|
+
# Confirm that the local copy is used
|
217
|
+
$ bundle install
|
218
|
+
|
219
|
+
# => Using good_job 0.1.0 from https://github.com/bensheldon/good_job.git (at /Users/You/Projects/good_job@dc57fb0)
|
220
|
+
```
|
221
|
+
|
222
|
+
## Releasing
|
223
|
+
|
224
|
+
Package maintainers can release this gem with the following [gem-release](https://github.com/svenfuchs/gem-release) command:
|
225
|
+
|
226
|
+
```bash
|
227
|
+
# Sign into rubygems
|
228
|
+
$ gem signin
|
229
|
+
|
230
|
+
# Update version number, changelog, and create git commit:
|
231
|
+
$ bundle exec rake commit_version[minor] # major,minor,patch
|
232
|
+
|
233
|
+
# ..and follow subsequent directions.
|
234
|
+
```
|
235
|
+
|
236
|
+
## Contributing
|
237
|
+
|
238
|
+
Contribution directions go here.
|
239
|
+
|
240
|
+
## License
|
241
|
+
|
242
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/exe/good_job
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
module ActiveJob
|
2
|
+
module QueueAdapters
|
3
|
+
class GoodJobAdapter < GoodJob::Adapter
|
4
|
+
def initialize(execution_mode: nil)
|
5
|
+
execution_mode = if execution_mode
|
6
|
+
execution_mode
|
7
|
+
elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
|
8
|
+
ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
|
9
|
+
elsif Rails.env.development? || Rails.env.test?
|
10
|
+
:inline
|
11
|
+
else
|
12
|
+
:external
|
13
|
+
end
|
14
|
+
|
15
|
+
super(execution_mode: execution_mode)
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
data/lib/good_job.rb
ADDED
@@ -0,0 +1,19 @@
|
|
1
|
+
require "rails"
|
2
|
+
require 'good_job/railtie'
|
3
|
+
|
4
|
+
require 'good_job/logging'
|
5
|
+
require 'good_job/lockable'
|
6
|
+
require 'good_job/job'
|
7
|
+
require 'good_job/scheduler'
|
8
|
+
require 'good_job/adapter'
|
9
|
+
require 'good_job/pg_locks'
|
10
|
+
require 'good_job/performer'
|
11
|
+
|
12
|
+
require 'active_job/queue_adapters/good_job_adapter'
|
13
|
+
|
14
|
+
module GoodJob
|
15
|
+
mattr_accessor :preserve_job_records, default: false
|
16
|
+
include Logging
|
17
|
+
|
18
|
+
ActiveSupport.run_load_hooks(:good_job, self)
|
19
|
+
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
module GoodJob
|
2
|
+
class Adapter
|
3
|
+
EXECUTION_MODES = [:inline, :external].freeze # TODO: async
|
4
|
+
|
5
|
+
def initialize(execution_mode: nil, inline: false)
|
6
|
+
if inline
|
7
|
+
ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
|
8
|
+
@execution_mode = :inline
|
9
|
+
elsif execution_mode
|
10
|
+
raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(execution_mode)
|
11
|
+
|
12
|
+
@execution_mode = execution_mode
|
13
|
+
else
|
14
|
+
@execution_mode = :external
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
def enqueue(active_job)
|
19
|
+
enqueue_at(active_job, nil)
|
20
|
+
end
|
21
|
+
|
22
|
+
def enqueue_at(active_job, timestamp)
|
23
|
+
good_job = GoodJob::Job.enqueue(
|
24
|
+
active_job,
|
25
|
+
scheduled_at: timestamp ? Time.zone.at(timestamp) : nil,
|
26
|
+
create_with_advisory_lock: execute_inline?
|
27
|
+
)
|
28
|
+
|
29
|
+
if execute_inline?
|
30
|
+
begin
|
31
|
+
good_job.perform
|
32
|
+
ensure
|
33
|
+
good_job.advisory_unlock
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
good_job
|
38
|
+
end
|
39
|
+
|
40
|
+
def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
|
41
|
+
nil
|
42
|
+
end
|
43
|
+
|
44
|
+
def execute_inline?
|
45
|
+
@execution_mode == :inline
|
46
|
+
end
|
47
|
+
|
48
|
+
def inline?
|
49
|
+
ActiveSupport::Deprecation.warn('GoodJob::Adapter::inline? is deprecated; use GoodJob::Adapter::execute_inline? instead')
|
50
|
+
execute_inline?
|
51
|
+
end
|
52
|
+
|
53
|
+
def execute_externally?
|
54
|
+
@execution_mode == :external
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
data/lib/good_job/cli.rb
ADDED
@@ -0,0 +1,91 @@
|
|
1
|
+
require 'thor'
|
2
|
+
|
3
|
+
module GoodJob
|
4
|
+
class CLI < Thor
|
5
|
+
RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
|
6
|
+
|
7
|
+
desc :start, "Start job worker"
|
8
|
+
method_option :max_threads,
|
9
|
+
type: :numeric,
|
10
|
+
desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
|
11
|
+
method_option :queues,
|
12
|
+
type: :string,
|
13
|
+
banner: "queue1,queue2",
|
14
|
+
desc: "Queues to work from. Separate multiple queues with commas (default: *)"
|
15
|
+
method_option :poll_interval,
|
16
|
+
type: :numeric,
|
17
|
+
desc: "Interval between polls for available jobs in seconds (default: 1)"
|
18
|
+
def start
|
19
|
+
require RAILS_ENVIRONMENT_RB
|
20
|
+
|
21
|
+
max_threads = (
|
22
|
+
options[:max_threads] ||
|
23
|
+
ENV['GOOD_JOB_MAX_THREADS'] ||
|
24
|
+
ENV['RAILS_MAX_THREADS'] ||
|
25
|
+
ActiveRecord::Base.connection_pool.size
|
26
|
+
).to_i
|
27
|
+
|
28
|
+
queue_names = (
|
29
|
+
options[:queues] ||
|
30
|
+
ENV['GOOD_JOB_QUEUES'] ||
|
31
|
+
'*'
|
32
|
+
).split(',').map(&:strip)
|
33
|
+
|
34
|
+
poll_interval = (
|
35
|
+
options[:poll_interval] ||
|
36
|
+
ENV['GOOD_JOB_POLL_INTERVAL']
|
37
|
+
).to_i
|
38
|
+
|
39
|
+
job_query = GoodJob::Job.all.priority_ordered
|
40
|
+
queue_names_without_all = queue_names.reject { |q| q == '*' }
|
41
|
+
job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
|
42
|
+
|
43
|
+
performer_method = if GoodJob.preserve_job_records
|
44
|
+
:perform_with_advisory_lock_and_preserve_job_records
|
45
|
+
else
|
46
|
+
:perform_with_advisory_lock_and_destroy_job_records
|
47
|
+
end
|
48
|
+
job_performer = GoodJob::Performer.new(job_query, performer_method)
|
49
|
+
|
50
|
+
$stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
|
51
|
+
|
52
|
+
timer_options = {}
|
53
|
+
timer_options[:execution_interval] = poll_interval if poll_interval.positive?
|
54
|
+
|
55
|
+
pool_options = {
|
56
|
+
max_threads: max_threads,
|
57
|
+
}
|
58
|
+
|
59
|
+
scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
|
60
|
+
|
61
|
+
@stop_good_job_executable = false
|
62
|
+
%w[INT TERM].each do |signal|
|
63
|
+
trap(signal) { @stop_good_job_executable = true }
|
64
|
+
end
|
65
|
+
|
66
|
+
Kernel.loop do
|
67
|
+
sleep 0.1
|
68
|
+
break if @stop_good_job_executable || scheduler.shutdown?
|
69
|
+
end
|
70
|
+
|
71
|
+
$stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
|
72
|
+
scheduler.shutdown
|
73
|
+
$stdout.puts "GoodJob's jobs finished, exiting..."
|
74
|
+
end
|
75
|
+
|
76
|
+
desc :cleanup_preserved_jobs, "Delete preserved job records"
|
77
|
+
method_option :before_seconds_ago,
|
78
|
+
type: :numeric,
|
79
|
+
default: 24 * 60 * 60,
|
80
|
+
desc: "Delete records finished more than this many seconds ago"
|
81
|
+
def cleanup_preserved_jobs
|
82
|
+
require RAILS_ENVIRONMENT_RB
|
83
|
+
|
84
|
+
timestamp = Time.current - options[:before_seconds_ago]
|
85
|
+
result = GoodJob::Job.finished(timestamp).delete_all
|
86
|
+
$stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
|
87
|
+
end
|
88
|
+
|
89
|
+
default_task :start
|
90
|
+
end
|
91
|
+
end
|
data/lib/good_job/job.rb
ADDED
@@ -0,0 +1,106 @@
|
|
1
|
+
module GoodJob
|
2
|
+
class Job < ActiveRecord::Base
|
3
|
+
include Lockable
|
4
|
+
|
5
|
+
PreviouslyPerformedError = Class.new(StandardError)
|
6
|
+
|
7
|
+
DEFAULT_QUEUE_NAME = 'default'.freeze
|
8
|
+
DEFAULT_PRIORITY = 0
|
9
|
+
|
10
|
+
self.table_name = 'good_jobs'.freeze
|
11
|
+
|
12
|
+
scope :unfinished, (lambda do
|
13
|
+
if column_names.include?('finished_at')
|
14
|
+
where(finished_at: nil)
|
15
|
+
else
|
16
|
+
ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
|
17
|
+
nil
|
18
|
+
end
|
19
|
+
end)
|
20
|
+
scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
|
21
|
+
scope :priority_ordered, -> { order(priority: :desc) }
|
22
|
+
scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
|
23
|
+
|
24
|
+
def self.perform_with_advisory_lock(destroy_after: !GoodJob.preserve_job_records)
|
25
|
+
good_job = nil
|
26
|
+
result = nil
|
27
|
+
error = nil
|
28
|
+
|
29
|
+
unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
|
30
|
+
good_job = good_jobs.first
|
31
|
+
break unless good_job
|
32
|
+
|
33
|
+
result, error = good_job.perform(destroy_after: destroy_after)
|
34
|
+
end
|
35
|
+
|
36
|
+
[good_job, result, error] if good_job
|
37
|
+
end
|
38
|
+
|
39
|
+
def self.perform_with_advisory_lock_and_preserve_job_records
|
40
|
+
perform_with_advisory_lock(destroy_after: false)
|
41
|
+
end
|
42
|
+
|
43
|
+
def self.perform_with_advisory_lock_and_destroy_job_records
|
44
|
+
perform_with_advisory_lock(destroy_after: true)
|
45
|
+
end
|
46
|
+
|
47
|
+
def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
|
48
|
+
good_job = nil
|
49
|
+
ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
|
50
|
+
good_job = GoodJob::Job.new(
|
51
|
+
queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
|
52
|
+
priority: active_job.priority || DEFAULT_PRIORITY,
|
53
|
+
serialized_params: active_job.serialize,
|
54
|
+
scheduled_at: scheduled_at || Time.current,
|
55
|
+
create_with_advisory_lock: create_with_advisory_lock
|
56
|
+
)
|
57
|
+
|
58
|
+
instrument_payload[:good_job] = good_job
|
59
|
+
|
60
|
+
good_job.save!
|
61
|
+
active_job.provider_job_id = good_job.id
|
62
|
+
end
|
63
|
+
|
64
|
+
good_job
|
65
|
+
end
|
66
|
+
|
67
|
+
def perform(destroy_after: true)
|
68
|
+
raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
|
69
|
+
|
70
|
+
result = nil
|
71
|
+
error = nil
|
72
|
+
|
73
|
+
ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
|
74
|
+
self.performed_at = Time.current
|
75
|
+
save! unless destroy_after
|
76
|
+
|
77
|
+
ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
|
78
|
+
params = serialized_params.merge(
|
79
|
+
"provider_job_id" => id
|
80
|
+
)
|
81
|
+
begin
|
82
|
+
result = ActiveJob::Base.execute(params)
|
83
|
+
rescue StandardError => e
|
84
|
+
error = e
|
85
|
+
end
|
86
|
+
end
|
87
|
+
|
88
|
+
if error.nil? && result.is_a?(Exception)
|
89
|
+
error = result
|
90
|
+
result = nil
|
91
|
+
end
|
92
|
+
|
93
|
+
error_message = "#{error.class}: #{error.message}" if error
|
94
|
+
self.error = error_message
|
95
|
+
self.finished_at = Time.current
|
96
|
+
|
97
|
+
if destroy_after
|
98
|
+
destroy!
|
99
|
+
else
|
100
|
+
save!
|
101
|
+
end
|
102
|
+
|
103
|
+
[result, error]
|
104
|
+
end
|
105
|
+
end
|
106
|
+
end
|
@@ -0,0 +1,106 @@
|
|
1
|
+
module GoodJob
|
2
|
+
module Lockable
|
3
|
+
extend ActiveSupport::Concern
|
4
|
+
|
5
|
+
RecordAlreadyAdvisoryLockedError = Class.new(StandardError)
|
6
|
+
|
7
|
+
included do
|
8
|
+
scope :advisory_lock, (lambda do
|
9
|
+
original_query = self
|
10
|
+
|
11
|
+
cte_table = Arel::Table.new(:rows)
|
12
|
+
composed_cte = Arel::Nodes::As.new(cte_table, original_query.select(primary_key).except(:limit).arel)
|
13
|
+
|
14
|
+
query = cte_table.project(cte_table[:id])
|
15
|
+
.with(composed_cte)
|
16
|
+
.where(Arel.sql(sanitize_sql_for_conditions(["pg_try_advisory_lock(('x'||substr(md5(:table_name || \"#{cte_table.name}\".\"#{primary_key}\"::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
|
17
|
+
|
18
|
+
limit = original_query.arel.ast.limit
|
19
|
+
query.limit = limit.value if limit.present?
|
20
|
+
|
21
|
+
unscoped.where(arel_table[:id].in(query)).merge(original_query.only(:order))
|
22
|
+
end)
|
23
|
+
|
24
|
+
scope :joins_advisory_locks, (lambda do
|
25
|
+
join_sql = <<~SQL
|
26
|
+
LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
|
27
|
+
AND pg_locks.objsubid = 1
|
28
|
+
AND pg_locks.classid = ('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(32)::int
|
29
|
+
AND pg_locks.objid = (('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(64) << 32)::bit(32)::int
|
30
|
+
SQL
|
31
|
+
|
32
|
+
joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
|
33
|
+
end)
|
34
|
+
|
35
|
+
scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
|
36
|
+
scope :advisory_locked, -> { joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
|
37
|
+
scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
|
38
|
+
|
39
|
+
attr_accessor :create_with_advisory_lock
|
40
|
+
|
41
|
+
after_create -> { advisory_lock }, if: :create_with_advisory_lock
|
42
|
+
end
|
43
|
+
|
44
|
+
class_methods do
|
45
|
+
def with_advisory_lock
|
46
|
+
raise ArgumentError, "Must provide a block" unless block_given?
|
47
|
+
|
48
|
+
records = advisory_lock.to_a
|
49
|
+
begin
|
50
|
+
yield(records)
|
51
|
+
ensure
|
52
|
+
records.each(&:advisory_unlock)
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def advisory_lock
|
58
|
+
query = <<~SQL
|
59
|
+
SELECT 1 AS one
|
60
|
+
WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
|
61
|
+
SQL
|
62
|
+
self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
|
63
|
+
end
|
64
|
+
|
65
|
+
def advisory_unlock
|
66
|
+
query = <<~SQL
|
67
|
+
SELECT 1 AS one
|
68
|
+
WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
|
69
|
+
SQL
|
70
|
+
self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
|
71
|
+
end
|
72
|
+
|
73
|
+
def advisory_lock!
|
74
|
+
result = advisory_lock
|
75
|
+
result || raise(RecordAlreadyAdvisoryLockedError)
|
76
|
+
end
|
77
|
+
|
78
|
+
def with_advisory_lock
|
79
|
+
raise ArgumentError, "Must provide a block" unless block_given?
|
80
|
+
|
81
|
+
advisory_lock!
|
82
|
+
yield
|
83
|
+
ensure
|
84
|
+
advisory_unlock unless $ERROR_INFO.is_a? RecordAlreadyAdvisoryLockedError
|
85
|
+
end
|
86
|
+
|
87
|
+
def advisory_locked?
|
88
|
+
self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
|
89
|
+
end
|
90
|
+
|
91
|
+
def owns_advisory_lock?
|
92
|
+
self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
|
93
|
+
end
|
94
|
+
|
95
|
+
def advisory_unlock!
|
96
|
+
advisory_unlock while advisory_locked?
|
97
|
+
end
|
98
|
+
|
99
|
+
private
|
100
|
+
|
101
|
+
def sanitize_sql_for_conditions(*args)
|
102
|
+
# Made public in Rails 5.2
|
103
|
+
self.class.send(:sanitize_sql_for_conditions, *args)
|
104
|
+
end
|
105
|
+
end
|
106
|
+
end
|
@@ -0,0 +1,70 @@
|
|
1
|
+
module GoodJob
|
2
|
+
module Logging
|
3
|
+
extend ActiveSupport::Concern
|
4
|
+
|
5
|
+
included do
|
6
|
+
cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
|
7
|
+
|
8
|
+
def self.tag_logger(*tags)
|
9
|
+
if logger.respond_to?(:tagged)
|
10
|
+
tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
|
11
|
+
logger.tagged(*tags) { yield }
|
12
|
+
else
|
13
|
+
yield
|
14
|
+
end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
class LogSubscriber < ActiveSupport::LogSubscriber
|
19
|
+
def create(event)
|
20
|
+
good_job = event.payload[:good_job]
|
21
|
+
|
22
|
+
info do
|
23
|
+
"Created GoodJob resource with id #{good_job.id}"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def timer_task_finished(event)
|
28
|
+
exception = event.payload[:error]
|
29
|
+
return unless exception
|
30
|
+
|
31
|
+
error do
|
32
|
+
"ERROR: #{exception}\n #{exception.backtrace}"
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
def job_finished(event)
|
37
|
+
exception = event.payload[:error]
|
38
|
+
return unless exception
|
39
|
+
|
40
|
+
error do
|
41
|
+
"ERROR: #{exception}\n #{exception.backtrace}"
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
def scheduler_start_shutdown(_event)
|
46
|
+
info do
|
47
|
+
"Shutting down scheduler..."
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
def scheduler_shutdown(_event)
|
52
|
+
info do
|
53
|
+
"Scheduler is shut down."
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
private
|
58
|
+
|
59
|
+
def logger
|
60
|
+
GoodJob.logger
|
61
|
+
end
|
62
|
+
|
63
|
+
def thread_name
|
64
|
+
Thread.current.name || Thread.current.object_id
|
65
|
+
end
|
66
|
+
end
|
67
|
+
end
|
68
|
+
end
|
69
|
+
|
70
|
+
GoodJob::Logging::LogSubscriber.attach_to :good_job
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module GoodJob
|
2
|
+
class PgLocks < ActiveRecord::Base
|
3
|
+
self.table_name = 'pg_locks'.freeze
|
4
|
+
|
5
|
+
# https://www.postgresql.org/docs/9.6/view-pg-locks.html
|
6
|
+
# Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
|
7
|
+
# A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
|
8
|
+
# The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
|
9
|
+
# Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
|
10
|
+
# The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
|
11
|
+
def self.advisory_lock_details
|
12
|
+
connection.select <<~SQL
|
13
|
+
SELECT *
|
14
|
+
FROM pg_locks
|
15
|
+
WHERE
|
16
|
+
locktype = 'advisory' AND
|
17
|
+
objsubid = 1
|
18
|
+
SQL
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,79 @@
|
|
1
|
+
require "concurrent/executor/thread_pool_executor"
|
2
|
+
require "concurrent/timer_task"
|
3
|
+
require "concurrent/utility/processor_counter"
|
4
|
+
|
5
|
+
module GoodJob
|
6
|
+
class Scheduler
|
7
|
+
DEFAULT_TIMER_OPTIONS = {
|
8
|
+
execution_interval: 1,
|
9
|
+
timeout_interval: 1,
|
10
|
+
run_now: true,
|
11
|
+
}.freeze
|
12
|
+
|
13
|
+
DEFAULT_POOL_OPTIONS = {
|
14
|
+
name: 'good_job',
|
15
|
+
min_threads: 0,
|
16
|
+
max_threads: Concurrent.processor_count,
|
17
|
+
auto_terminate: true,
|
18
|
+
idletime: 60,
|
19
|
+
max_queue: 0,
|
20
|
+
fallback_policy: :abort, # shouldn't matter -- 0 max queue
|
21
|
+
}.freeze
|
22
|
+
|
23
|
+
def initialize(performer, timer_options: {}, pool_options: {})
|
24
|
+
raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
|
25
|
+
|
26
|
+
@performer = performer
|
27
|
+
@pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
|
28
|
+
@timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
|
29
|
+
idle_threads = @pool.max_length - @pool.length
|
30
|
+
create_thread if idle_threads.positive?
|
31
|
+
end
|
32
|
+
@timer.add_observer(self, :timer_observer)
|
33
|
+
@timer.execute
|
34
|
+
end
|
35
|
+
|
36
|
+
def execute
|
37
|
+
end
|
38
|
+
|
39
|
+
def shutdown(wait: true)
|
40
|
+
@_shutdown = true
|
41
|
+
|
42
|
+
ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
|
43
|
+
ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
|
44
|
+
if @timer.running?
|
45
|
+
@timer.shutdown
|
46
|
+
@timer.wait_for_termination if wait
|
47
|
+
end
|
48
|
+
|
49
|
+
if @pool.running?
|
50
|
+
@pool.shutdown
|
51
|
+
@pool.wait_for_termination if wait
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
55
|
+
|
56
|
+
def shutdown?
|
57
|
+
@_shutdown
|
58
|
+
end
|
59
|
+
|
60
|
+
def create_thread
|
61
|
+
future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
|
62
|
+
output = nil
|
63
|
+
Rails.application.executor.wrap { output = performer.next }
|
64
|
+
output
|
65
|
+
end
|
66
|
+
future.add_observer(self, :task_observer)
|
67
|
+
future.execute
|
68
|
+
end
|
69
|
+
|
70
|
+
def timer_observer(time, executed_task, thread_error)
|
71
|
+
ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
|
72
|
+
end
|
73
|
+
|
74
|
+
def task_observer(time, output, thread_error)
|
75
|
+
ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
|
76
|
+
create_thread if output
|
77
|
+
end
|
78
|
+
end
|
79
|
+
end
|
metadata
ADDED
@@ -0,0 +1,285 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: good_job
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 1.0.1
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Ben Sheldon
|
8
|
+
autorequire:
|
9
|
+
bindir: exe
|
10
|
+
cert_chain: []
|
11
|
+
date: 2020-07-22 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: concurrent-ruby
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: 1.0.2
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: 1.0.2
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: pg
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 1.0.0
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - ">="
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: 1.0.0
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: rails
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - ">="
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: 5.1.0
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: 5.1.0
|
55
|
+
- !ruby/object:Gem::Dependency
|
56
|
+
name: thor
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: 0.14.1
|
62
|
+
type: :runtime
|
63
|
+
prerelease: false
|
64
|
+
version_requirements: !ruby/object:Gem::Requirement
|
65
|
+
requirements:
|
66
|
+
- - ">="
|
67
|
+
- !ruby/object:Gem::Version
|
68
|
+
version: 0.14.1
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: appraisal
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - ">="
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '0'
|
76
|
+
type: :development
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - ">="
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '0'
|
83
|
+
- !ruby/object:Gem::Dependency
|
84
|
+
name: database_cleaner
|
85
|
+
requirement: !ruby/object:Gem::Requirement
|
86
|
+
requirements:
|
87
|
+
- - ">="
|
88
|
+
- !ruby/object:Gem::Version
|
89
|
+
version: '0'
|
90
|
+
type: :development
|
91
|
+
prerelease: false
|
92
|
+
version_requirements: !ruby/object:Gem::Requirement
|
93
|
+
requirements:
|
94
|
+
- - ">="
|
95
|
+
- !ruby/object:Gem::Version
|
96
|
+
version: '0'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: foreman
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
type: :development
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - ">="
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
111
|
+
- !ruby/object:Gem::Dependency
|
112
|
+
name: gem-release
|
113
|
+
requirement: !ruby/object:Gem::Requirement
|
114
|
+
requirements:
|
115
|
+
- - ">="
|
116
|
+
- !ruby/object:Gem::Version
|
117
|
+
version: '0'
|
118
|
+
type: :development
|
119
|
+
prerelease: false
|
120
|
+
version_requirements: !ruby/object:Gem::Requirement
|
121
|
+
requirements:
|
122
|
+
- - ">="
|
123
|
+
- !ruby/object:Gem::Version
|
124
|
+
version: '0'
|
125
|
+
- !ruby/object:Gem::Dependency
|
126
|
+
name: github_changelog_generator
|
127
|
+
requirement: !ruby/object:Gem::Requirement
|
128
|
+
requirements:
|
129
|
+
- - ">="
|
130
|
+
- !ruby/object:Gem::Version
|
131
|
+
version: '0'
|
132
|
+
type: :development
|
133
|
+
prerelease: false
|
134
|
+
version_requirements: !ruby/object:Gem::Requirement
|
135
|
+
requirements:
|
136
|
+
- - ">="
|
137
|
+
- !ruby/object:Gem::Version
|
138
|
+
version: '0'
|
139
|
+
- !ruby/object:Gem::Dependency
|
140
|
+
name: pry
|
141
|
+
requirement: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
type: :development
|
147
|
+
prerelease: false
|
148
|
+
version_requirements: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
153
|
+
- !ruby/object:Gem::Dependency
|
154
|
+
name: rspec-rails
|
155
|
+
requirement: !ruby/object:Gem::Requirement
|
156
|
+
requirements:
|
157
|
+
- - ">="
|
158
|
+
- !ruby/object:Gem::Version
|
159
|
+
version: '0'
|
160
|
+
type: :development
|
161
|
+
prerelease: false
|
162
|
+
version_requirements: !ruby/object:Gem::Requirement
|
163
|
+
requirements:
|
164
|
+
- - ">="
|
165
|
+
- !ruby/object:Gem::Version
|
166
|
+
version: '0'
|
167
|
+
- !ruby/object:Gem::Dependency
|
168
|
+
name: rubocop
|
169
|
+
requirement: !ruby/object:Gem::Requirement
|
170
|
+
requirements:
|
171
|
+
- - ">="
|
172
|
+
- !ruby/object:Gem::Version
|
173
|
+
version: '0'
|
174
|
+
type: :development
|
175
|
+
prerelease: false
|
176
|
+
version_requirements: !ruby/object:Gem::Requirement
|
177
|
+
requirements:
|
178
|
+
- - ">="
|
179
|
+
- !ruby/object:Gem::Version
|
180
|
+
version: '0'
|
181
|
+
- !ruby/object:Gem::Dependency
|
182
|
+
name: rubocop-performance
|
183
|
+
requirement: !ruby/object:Gem::Requirement
|
184
|
+
requirements:
|
185
|
+
- - ">="
|
186
|
+
- !ruby/object:Gem::Version
|
187
|
+
version: '0'
|
188
|
+
type: :development
|
189
|
+
prerelease: false
|
190
|
+
version_requirements: !ruby/object:Gem::Requirement
|
191
|
+
requirements:
|
192
|
+
- - ">="
|
193
|
+
- !ruby/object:Gem::Version
|
194
|
+
version: '0'
|
195
|
+
- !ruby/object:Gem::Dependency
|
196
|
+
name: rubocop-rails
|
197
|
+
requirement: !ruby/object:Gem::Requirement
|
198
|
+
requirements:
|
199
|
+
- - ">="
|
200
|
+
- !ruby/object:Gem::Version
|
201
|
+
version: '0'
|
202
|
+
type: :development
|
203
|
+
prerelease: false
|
204
|
+
version_requirements: !ruby/object:Gem::Requirement
|
205
|
+
requirements:
|
206
|
+
- - ">="
|
207
|
+
- !ruby/object:Gem::Version
|
208
|
+
version: '0'
|
209
|
+
- !ruby/object:Gem::Dependency
|
210
|
+
name: rubocop-rspec
|
211
|
+
requirement: !ruby/object:Gem::Requirement
|
212
|
+
requirements:
|
213
|
+
- - ">="
|
214
|
+
- !ruby/object:Gem::Version
|
215
|
+
version: '0'
|
216
|
+
type: :development
|
217
|
+
prerelease: false
|
218
|
+
version_requirements: !ruby/object:Gem::Requirement
|
219
|
+
requirements:
|
220
|
+
- - ">="
|
221
|
+
- !ruby/object:Gem::Version
|
222
|
+
version: '0'
|
223
|
+
description: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
|
224
|
+
email:
|
225
|
+
- bensheldon@gmail.com
|
226
|
+
executables:
|
227
|
+
- good_job
|
228
|
+
extensions: []
|
229
|
+
extra_rdoc_files:
|
230
|
+
- README.md
|
231
|
+
- CHANGELOG.md
|
232
|
+
- LICENSE.txt
|
233
|
+
files:
|
234
|
+
- CHANGELOG.md
|
235
|
+
- LICENSE.txt
|
236
|
+
- README.md
|
237
|
+
- exe/good_job
|
238
|
+
- lib/active_job/queue_adapters/good_job_adapter.rb
|
239
|
+
- lib/good_job.rb
|
240
|
+
- lib/good_job/adapter.rb
|
241
|
+
- lib/good_job/cli.rb
|
242
|
+
- lib/good_job/job.rb
|
243
|
+
- lib/good_job/lockable.rb
|
244
|
+
- lib/good_job/logging.rb
|
245
|
+
- lib/good_job/performer.rb
|
246
|
+
- lib/good_job/pg_locks.rb
|
247
|
+
- lib/good_job/railtie.rb
|
248
|
+
- lib/good_job/scheduler.rb
|
249
|
+
- lib/good_job/version.rb
|
250
|
+
homepage: https://github.com/bensheldon/good_job
|
251
|
+
licenses:
|
252
|
+
- MIT
|
253
|
+
metadata:
|
254
|
+
bug_tracker_uri: https://github.com/bensheldon/good_job/issues
|
255
|
+
changelog_uri: https://github.com/bensheldon/good_job/blob/master/CHANGELOG.md
|
256
|
+
documentation_uri: https://rdoc.info/github/bensheldon/good_job
|
257
|
+
homepage_uri: https://github.com/bensheldon/good_job
|
258
|
+
source_code_uri: https://github.com/bensheldon/good_job
|
259
|
+
post_install_message:
|
260
|
+
rdoc_options:
|
261
|
+
- "--title"
|
262
|
+
- GoodJob - a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
|
263
|
+
- "--main"
|
264
|
+
- README.md
|
265
|
+
- "--line-numbers"
|
266
|
+
- "--inline-source"
|
267
|
+
- "--quiet"
|
268
|
+
require_paths:
|
269
|
+
- lib
|
270
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
271
|
+
requirements:
|
272
|
+
- - ">="
|
273
|
+
- !ruby/object:Gem::Version
|
274
|
+
version: 2.4.0
|
275
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
276
|
+
requirements:
|
277
|
+
- - ">="
|
278
|
+
- !ruby/object:Gem::Version
|
279
|
+
version: '0'
|
280
|
+
requirements: []
|
281
|
+
rubygems_version: 3.0.3
|
282
|
+
signing_key:
|
283
|
+
specification_version: 4
|
284
|
+
summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
|
285
|
+
test_files: []
|