good_job 0.2.1 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +56 -2
- data/README.md +53 -15
- data/lib/active_job/queue_adapters/good_job_adapter.rb +13 -0
- data/lib/good_job.rb +3 -2
- data/lib/good_job/adapter.rb +20 -24
- data/lib/good_job/cli.rb +56 -3
- data/lib/good_job/job.rb +56 -0
- data/lib/good_job/lockable.rb +73 -74
- data/lib/good_job/logging.rb +12 -0
- data/lib/good_job/pg_locks.rb +21 -0
- data/lib/good_job/scheduler.rb +33 -45
- data/lib/good_job/version.rb +1 -1
- metadata +33 -5
- data/lib/good_job/inline_scheduler.rb +0 -10
- data/lib/good_job/job_wrapper.rb +0 -16
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: c20439024a46084c22e46bb42e38589687c2ab2cea47d90705abc4fa042ad449
|
4
|
+
data.tar.gz: e5ea8e7fa607a10fd936c62623a39d32b891ef629636e0b04fb3d36aa7ab1f3f
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4af34ce400c409a61892c9d35693f69e023b5dbd3d54cacfd6d9b044aaf481c616e3fdb9da53470f79919830a916a08f67550ce807eda1c59a43fa1da4092f07
|
7
|
+
data.tar.gz: 81304be6bc9e6925b16fa030d2031377bd90bac894b4c8792cfe5e02ff81677bf04d476b1a3435e951a804e6d284e4d8aa7452c6090969f841dbd6e59a9c301c
|
data/CHANGELOG.md
CHANGED
@@ -1,8 +1,62 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
-
## [
|
3
|
+
## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
|
4
4
|
|
5
|
-
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.5.0...v0.6.0)
|
6
|
+
|
7
|
+
**Closed issues:**
|
8
|
+
|
9
|
+
- Improve the command line options [\#32](https://github.com/bensheldon/good_job/issues/32)
|
10
|
+
- Allow config.active\_job.queue\_adapter = :good\_job to work [\#5](https://github.com/bensheldon/good_job/issues/5)
|
11
|
+
|
12
|
+
**Merged pull requests:**
|
13
|
+
|
14
|
+
- Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
|
15
|
+
- Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
|
16
|
+
- Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
|
17
|
+
- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
|
18
|
+
- Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
|
19
|
+
|
20
|
+
## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
|
21
|
+
|
22
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.4.0...v0.5.0)
|
23
|
+
|
24
|
+
**Merged pull requests:**
|
25
|
+
|
26
|
+
- Update development Ruby to 2.6.6 and gems [\#29](https://github.com/bensheldon/good_job/pull/29) ([bensheldon](https://github.com/bensheldon))
|
27
|
+
|
28
|
+
## [v0.4.0](https://github.com/bensheldon/good_job/tree/v0.4.0) (2020-03-31)
|
29
|
+
|
30
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.3.0...v0.4.0)
|
31
|
+
|
32
|
+
**Merged pull requests:**
|
33
|
+
|
34
|
+
- Improve ActiveRecord usage for advisory locking [\#24](https://github.com/bensheldon/good_job/pull/24) ([bensheldon](https://github.com/bensheldon))
|
35
|
+
- Remove support for Rails 5.1 [\#23](https://github.com/bensheldon/good_job/pull/23) ([bensheldon](https://github.com/bensheldon))
|
36
|
+
|
37
|
+
## [v0.3.0](https://github.com/bensheldon/good_job/tree/v0.3.0) (2020-03-22)
|
38
|
+
|
39
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.2...v0.3.0)
|
40
|
+
|
41
|
+
**Merged pull requests:**
|
42
|
+
|
43
|
+
- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
|
44
|
+
- Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
|
45
|
+
- Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
|
46
|
+
- Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
|
47
|
+
|
48
|
+
## [v0.2.2](https://github.com/bensheldon/good_job/tree/v0.2.2) (2020-03-08)
|
49
|
+
|
50
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...v0.2.2)
|
51
|
+
|
52
|
+
**Merged pull requests:**
|
53
|
+
|
54
|
+
- Gracefully shutdown Scheduler when executable receives TERM or INT [\#17](https://github.com/bensheldon/good_job/pull/17) ([bensheldon](https://github.com/bensheldon))
|
55
|
+
- Update Appraisals [\#16](https://github.com/bensheldon/good_job/pull/16) ([bensheldon](https://github.com/bensheldon))
|
56
|
+
|
57
|
+
## [v0.2.1](https://github.com/bensheldon/good_job/tree/v0.2.1) (2020-03-07)
|
58
|
+
|
59
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.0...v0.2.1)
|
6
60
|
|
7
61
|
**Merged pull requests:**
|
8
62
|
|
data/README.md
CHANGED
@@ -14,12 +14,12 @@ Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Q
|
|
14
14
|
Add this line to your application's Gemfile:
|
15
15
|
|
16
16
|
```ruby
|
17
|
-
gem 'good_job'
|
17
|
+
gem 'good_job'
|
18
18
|
```
|
19
19
|
|
20
20
|
And then execute:
|
21
21
|
```bash
|
22
|
-
$ bundle
|
22
|
+
$ bundle install
|
23
23
|
```
|
24
24
|
|
25
25
|
## Usage
|
@@ -56,17 +56,51 @@ $ bundle
|
|
56
56
|
|
57
57
|
1. Configure the ActiveJob adapter:
|
58
58
|
```ruby
|
59
|
-
# config/
|
60
|
-
config.active_job.queue_adapter =
|
61
|
-
|
59
|
+
# config/application.rb
|
60
|
+
config.active_job.queue_adapter = :good_job
|
61
|
+
```
|
62
|
+
|
63
|
+
By default, using `:good_job` is equivalent to manually configuring the adapter:
|
64
|
+
|
65
|
+
```ruby
|
62
66
|
# config/environments/development.rb
|
63
67
|
config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
|
68
|
+
|
69
|
+
# config/environments/test.rb
|
70
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
|
71
|
+
|
72
|
+
# config/environments/production.rb
|
73
|
+
config.active_job.queue_adapter = GoodJob::Adapter.new
|
64
74
|
```
|
65
75
|
|
66
76
|
1. In production, the scheduler is designed to run in its own process:
|
67
77
|
```bash
|
68
78
|
$ bundle exec good_job
|
69
79
|
```
|
80
|
+
|
81
|
+
Configuration options available with `help`:
|
82
|
+
```bash
|
83
|
+
$ bundle exec good_job help start
|
84
|
+
|
85
|
+
# Usage:
|
86
|
+
# good_job start
|
87
|
+
#
|
88
|
+
# Options:
|
89
|
+
# [--max-threads=N] # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
|
90
|
+
# [--queues=queue1,queue2] # Queues to work from. Separate multiple queues with commas (default: *)
|
91
|
+
# [--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
|
92
|
+
```
|
93
|
+
|
94
|
+
### Configuring Job Execution Threads
|
95
|
+
|
96
|
+
GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
|
97
|
+
|
98
|
+
- Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._
|
99
|
+
- The maximum number of GoodJob threads can be configured, in decreasing precedence:
|
100
|
+
1. `$ bundle exec good_job --max_threads 4`
|
101
|
+
2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
|
102
|
+
3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
|
103
|
+
4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
|
70
104
|
|
71
105
|
## Development
|
72
106
|
|
@@ -83,6 +117,17 @@ $ bin/setup_test
|
|
83
117
|
$ bin/rspec
|
84
118
|
```
|
85
119
|
|
120
|
+
This gem uses Appraisal to run tests against multiple versions of Rails:
|
121
|
+
|
122
|
+
```bash
|
123
|
+
# Install Appraisal(s) gemfiles
|
124
|
+
$ bundle exec appraisal
|
125
|
+
|
126
|
+
# Run tests
|
127
|
+
$ bundle exec appraisal bin/rspec
|
128
|
+
|
129
|
+
```
|
130
|
+
|
86
131
|
For developing locally within another Ruby on Rails project:
|
87
132
|
|
88
133
|
```bash
|
@@ -103,17 +148,10 @@ Package maintainers can release this gem with the following [gem-release](https:
|
|
103
148
|
# Sign into rubygems
|
104
149
|
$ gem signin
|
105
150
|
|
106
|
-
#
|
107
|
-
$
|
108
|
-
|
109
|
-
# Update the changelog
|
110
|
-
$ bundle exec rake changelog
|
111
|
-
|
112
|
-
# Commit the version and changelog to git
|
113
|
-
$ bundle exec rake commit_version
|
151
|
+
# Update version number, changelog, and create git commit:
|
152
|
+
$ bundle exec rake commit_version[minor] # major,minor,patch
|
114
153
|
|
115
|
-
#
|
116
|
-
$ gem release
|
154
|
+
# ..and follow subsequent directions.
|
117
155
|
```
|
118
156
|
|
119
157
|
## Contributing
|
data/lib/good_job.rb
CHANGED
@@ -4,10 +4,11 @@ require 'good_job/railtie'
|
|
4
4
|
require 'good_job/logging'
|
5
5
|
require 'good_job/lockable'
|
6
6
|
require 'good_job/job'
|
7
|
-
require 'good_job/inline_scheduler'
|
8
7
|
require "good_job/scheduler"
|
9
|
-
require "good_job/job_wrapper"
|
10
8
|
require 'good_job/adapter'
|
9
|
+
require 'good_job/pg_locks'
|
10
|
+
|
11
|
+
require 'active_job/queue_adapters/good_job_adapter'
|
11
12
|
|
12
13
|
module GoodJob
|
13
14
|
include Logging
|
data/lib/good_job/adapter.rb
CHANGED
@@ -1,41 +1,37 @@
|
|
1
1
|
module GoodJob
|
2
2
|
class Adapter
|
3
|
-
def initialize(
|
4
|
-
@
|
5
|
-
@scheduler = InlineScheduler.new if inline?
|
3
|
+
def initialize(inline: false)
|
4
|
+
@inline = inline
|
6
5
|
end
|
7
6
|
|
8
|
-
def enqueue(
|
9
|
-
enqueue_at(
|
7
|
+
def enqueue(active_job)
|
8
|
+
enqueue_at(active_job, nil)
|
10
9
|
end
|
11
10
|
|
12
|
-
def enqueue_at(
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
@scheduler.enqueue(good_job) if inline?
|
11
|
+
def enqueue_at(active_job, timestamp)
|
12
|
+
good_job = GoodJob::Job.enqueue(
|
13
|
+
active_job,
|
14
|
+
scheduled_at: timestamp ? Time.at(timestamp) : nil,
|
15
|
+
create_with_advisory_lock: inline?
|
16
|
+
)
|
17
|
+
|
18
|
+
if inline?
|
19
|
+
begin
|
20
|
+
good_job.perform
|
21
|
+
ensure
|
22
|
+
good_job.advisory_unlock
|
23
|
+
end
|
26
24
|
end
|
27
25
|
|
28
26
|
good_job
|
29
27
|
end
|
30
28
|
|
31
|
-
def shutdown(wait: true)
|
32
|
-
|
29
|
+
def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
|
30
|
+
nil
|
33
31
|
end
|
34
32
|
|
35
|
-
private
|
36
|
-
|
37
33
|
def inline?
|
38
|
-
@
|
34
|
+
@inline
|
39
35
|
end
|
40
36
|
end
|
41
37
|
end
|
data/lib/good_job/cli.rb
CHANGED
@@ -4,15 +4,68 @@ module GoodJob
|
|
4
4
|
class CLI < Thor
|
5
5
|
RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
|
6
6
|
|
7
|
-
desc :start, "Start
|
7
|
+
desc :start, "Start job worker"
|
8
|
+
method_option :max_threads,
|
9
|
+
type: :numeric,
|
10
|
+
desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
|
11
|
+
method_option :queues,
|
12
|
+
type: :string,
|
13
|
+
banner: "queue1,queue2",
|
14
|
+
desc: "Queues to work from. Separate multiple queues with commas (default: *)"
|
15
|
+
method_option :poll_interval,
|
16
|
+
type: :numeric,
|
17
|
+
desc: "Interval between polls for available jobs in seconds (default: 1)"
|
8
18
|
def start
|
9
19
|
require RAILS_ENVIRONMENT_RB
|
10
20
|
|
11
|
-
|
21
|
+
max_threads = (
|
22
|
+
options[:max_threads] ||
|
23
|
+
ENV['GOOD_JOB_MAX_THREADS'] ||
|
24
|
+
ENV['RAILS_MAX_THREADS'] ||
|
25
|
+
ActiveRecord::Base.connection_pool.size
|
26
|
+
).to_i
|
27
|
+
|
28
|
+
queue_names = (
|
29
|
+
options[:queues] ||
|
30
|
+
ENV['GOOD_JOB_QUEUES'] ||
|
31
|
+
'*'
|
32
|
+
).split(',').map(&:strip)
|
33
|
+
|
34
|
+
poll_interval = (
|
35
|
+
options[:poll_interval] ||
|
36
|
+
ENV['GOOD_JOB_POLL_INTERVAL']
|
37
|
+
).to_i
|
38
|
+
|
39
|
+
job_query = GoodJob::Job.all
|
40
|
+
queue_names_without_all = queue_names.reject { |q| q == '*' }
|
41
|
+
job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
|
42
|
+
|
43
|
+
job_performer = job_query.only_scheduled.priority_ordered.to_performer
|
44
|
+
|
45
|
+
$stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
|
46
|
+
|
47
|
+
timer_options = {}
|
48
|
+
timer_options[:execution_interval] = poll_interval if poll_interval.positive?
|
49
|
+
|
50
|
+
pool_options = {
|
51
|
+
max_threads: max_threads,
|
52
|
+
}
|
53
|
+
|
54
|
+
scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
|
55
|
+
|
56
|
+
@stop_good_job_executable = false
|
57
|
+
%w[INT TERM].each do |signal|
|
58
|
+
trap(signal) { @stop_good_job_executable = true }
|
59
|
+
end
|
12
60
|
|
13
61
|
Kernel.loop do
|
14
|
-
sleep 1
|
62
|
+
sleep 0.1
|
63
|
+
break if @stop_good_job_executable || scheduler.shutdown?
|
15
64
|
end
|
65
|
+
|
66
|
+
$stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
|
67
|
+
scheduler.shutdown
|
68
|
+
$stdout.puts "GoodJob's jobs finished, exiting..."
|
16
69
|
end
|
17
70
|
|
18
71
|
default_task :start
|
data/lib/good_job/job.rb
CHANGED
@@ -1,6 +1,62 @@
|
|
1
1
|
module GoodJob
|
2
2
|
class Job < ActiveRecord::Base
|
3
3
|
include Lockable
|
4
|
+
|
4
5
|
self.table_name = 'good_jobs'
|
6
|
+
|
7
|
+
scope :only_scheduled, -> { where("scheduled_at < ?", Time.current).or(where(scheduled_at: nil)) }
|
8
|
+
scope :priority_ordered, -> { order(priority: :desc) }
|
9
|
+
scope :to_performer, -> { Performer.new(self) }
|
10
|
+
|
11
|
+
class Performer
|
12
|
+
def initialize(query)
|
13
|
+
@query = query
|
14
|
+
end
|
15
|
+
|
16
|
+
def next
|
17
|
+
good_job = nil
|
18
|
+
|
19
|
+
@query.limit(1).with_advisory_lock do |good_jobs|
|
20
|
+
good_job = good_jobs.first
|
21
|
+
break unless good_job
|
22
|
+
|
23
|
+
good_job.perform
|
24
|
+
end
|
25
|
+
|
26
|
+
good_job
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
|
31
|
+
good_job = nil
|
32
|
+
ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
|
33
|
+
good_job = GoodJob::Job.new(
|
34
|
+
queue_name: active_job.queue_name,
|
35
|
+
priority: active_job.priority,
|
36
|
+
serialized_params: active_job.serialize,
|
37
|
+
scheduled_at: scheduled_at,
|
38
|
+
create_with_advisory_lock: create_with_advisory_lock
|
39
|
+
)
|
40
|
+
|
41
|
+
instrument_payload[:good_job] = good_job
|
42
|
+
|
43
|
+
good_job.save!
|
44
|
+
active_job.provider_job_id = good_job.id
|
45
|
+
end
|
46
|
+
|
47
|
+
good_job
|
48
|
+
end
|
49
|
+
|
50
|
+
def perform
|
51
|
+
ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
|
52
|
+
ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
|
53
|
+
params = serialized_params.merge(
|
54
|
+
"provider_job_id" => id
|
55
|
+
)
|
56
|
+
ActiveJob::Base.execute(params)
|
57
|
+
|
58
|
+
destroy!
|
59
|
+
end
|
60
|
+
end
|
5
61
|
end
|
6
62
|
end
|
data/lib/good_job/lockable.rb
CHANGED
@@ -5,98 +5,97 @@ module GoodJob
|
|
5
5
|
RecordAlreadyAdvisoryLockedError = Class.new(StandardError)
|
6
6
|
|
7
7
|
included do
|
8
|
+
scope :advisory_lock, (lambda do
|
9
|
+
original_query = self
|
10
|
+
|
11
|
+
cte_table = Arel::Table.new(:rows)
|
12
|
+
composed_cte = Arel::Nodes::As.new(cte_table, original_query.select(primary_key).except(:limit).arel)
|
13
|
+
|
14
|
+
query = cte_table.project(cte_table[:id])
|
15
|
+
.with(composed_cte)
|
16
|
+
.where(Arel.sql(sanitize_sql_for_conditions(["pg_try_advisory_lock(('x'||substr(md5(:table_name || \"#{cte_table.name}\".\"#{primary_key}\"::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
|
17
|
+
|
18
|
+
limit = original_query.arel.ast.limit
|
19
|
+
query.limit = limit.value if limit.present?
|
20
|
+
|
21
|
+
unscoped.where(arel_table[:id].in(query)).merge(original_query.only(:order))
|
22
|
+
end)
|
23
|
+
|
8
24
|
scope :joins_advisory_locks, (lambda do
|
9
|
-
|
25
|
+
join_sql = <<~SQL
|
10
26
|
LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
|
11
27
|
AND pg_locks.objsubid = 1
|
12
|
-
AND pg_locks.classid = ('x'||substr(md5(
|
13
|
-
AND pg_locks.objid = (('x'||substr(md5(
|
28
|
+
AND pg_locks.classid = ('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(32)::int
|
29
|
+
AND pg_locks.objid = (('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(64) << 32)::bit(32)::int
|
14
30
|
SQL
|
31
|
+
|
32
|
+
joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
|
15
33
|
end)
|
16
34
|
|
17
35
|
scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
|
18
|
-
scope :
|
19
|
-
|
20
|
-
pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
|
21
|
-
SQL
|
22
|
-
end)
|
36
|
+
scope :advisory_locked, -> { joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
|
37
|
+
scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
|
23
38
|
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
SELECT rows.id
|
28
|
-
FROM rows
|
29
|
-
WHERE pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
|
30
|
-
SQL
|
31
|
-
end
|
32
|
-
# private_class_method :first_advisory_locked_row
|
33
|
-
|
34
|
-
# https://www.postgresql.org/docs/9.6/view-pg-locks.html
|
35
|
-
# Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
|
36
|
-
# A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
|
37
|
-
# The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
|
38
|
-
# Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
|
39
|
-
# The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
|
40
|
-
def self.advisory_lock_details
|
41
|
-
connection.select("SELECT * FROM pg_locks WHERE locktype = 'advisory' AND objsubid = 1")
|
42
|
-
end
|
39
|
+
attr_accessor :create_with_advisory_lock
|
40
|
+
after_create -> { advisory_lock }, if: :create_with_advisory_lock
|
41
|
+
end
|
43
42
|
|
44
|
-
|
45
|
-
|
43
|
+
class_methods do
|
44
|
+
def with_advisory_lock(&block)
|
45
|
+
records = advisory_lock.to_a
|
46
|
+
begin
|
47
|
+
block.call(records)
|
48
|
+
ensure
|
49
|
+
records.each(&:advisory_unlock)
|
50
|
+
end
|
46
51
|
end
|
52
|
+
end
|
47
53
|
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
54
|
+
def advisory_lock
|
55
|
+
query = <<~SQL
|
56
|
+
SELECT 1 AS one
|
57
|
+
WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
|
58
|
+
SQL
|
59
|
+
self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
|
60
|
+
end
|
52
61
|
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
62
|
+
def advisory_unlock
|
63
|
+
query = <<~SQL
|
64
|
+
SELECT 1 AS one
|
65
|
+
WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
|
66
|
+
SQL
|
67
|
+
self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
|
68
|
+
end
|
60
69
|
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
WHERE
|
66
|
-
locktype = 'advisory'
|
67
|
-
AND objsubid = 1
|
68
|
-
AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
|
69
|
-
AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
|
70
|
-
SQL
|
71
|
-
end
|
70
|
+
def advisory_lock!
|
71
|
+
result = advisory_lock
|
72
|
+
result || raise(RecordAlreadyAdvisoryLockedError)
|
73
|
+
end
|
72
74
|
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
|
77
|
-
|
78
|
-
|
79
|
-
AND objsubid = 1
|
80
|
-
AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
|
81
|
-
AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
|
82
|
-
AND pid = pg_backend_pid()
|
83
|
-
SQL
|
84
|
-
end
|
75
|
+
def with_advisory_lock
|
76
|
+
advisory_lock!
|
77
|
+
yield
|
78
|
+
ensure
|
79
|
+
advisory_unlock unless $ERROR_INFO.is_a? RecordAlreadyAdvisoryLockedError
|
80
|
+
end
|
85
81
|
|
86
|
-
|
87
|
-
|
88
|
-
|
82
|
+
def advisory_locked?
|
83
|
+
self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
|
84
|
+
end
|
89
85
|
|
90
|
-
|
91
|
-
|
92
|
-
|
86
|
+
def owns_advisory_lock?
|
87
|
+
self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
|
88
|
+
end
|
93
89
|
|
94
|
-
|
90
|
+
def advisory_unlock!
|
91
|
+
advisory_unlock while advisory_locked?
|
92
|
+
end
|
95
93
|
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
94
|
+
private
|
95
|
+
|
96
|
+
def sanitize_sql_for_conditions(*args)
|
97
|
+
# Made public in Rails 5.2
|
98
|
+
self.class.send(:sanitize_sql_for_conditions, *args)
|
100
99
|
end
|
101
100
|
end
|
102
101
|
end
|
data/lib/good_job/logging.rb
CHANGED
@@ -42,6 +42,18 @@ module GoodJob
|
|
42
42
|
end
|
43
43
|
end
|
44
44
|
|
45
|
+
def scheduler_start_shutdown(_event)
|
46
|
+
info do
|
47
|
+
"Shutting down scheduler..."
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
def scheduler_shutdown(_event)
|
52
|
+
info do
|
53
|
+
"Scheduler is shut down."
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
45
57
|
private
|
46
58
|
|
47
59
|
def logger
|
@@ -0,0 +1,21 @@
|
|
1
|
+
module GoodJob
|
2
|
+
class PgLocks < ActiveRecord::Base
|
3
|
+
self.table_name = 'pg_locks'.freeze
|
4
|
+
|
5
|
+
# https://www.postgresql.org/docs/9.6/view-pg-locks.html
|
6
|
+
# Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
|
7
|
+
# A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
|
8
|
+
# The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
|
9
|
+
# Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
|
10
|
+
# The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
|
11
|
+
def self.advisory_lock_details
|
12
|
+
connection.select <<~SQL
|
13
|
+
SELECT *
|
14
|
+
FROM pg_locks
|
15
|
+
WHERE
|
16
|
+
locktype = 'advisory' AND
|
17
|
+
objsubid = 1
|
18
|
+
SQL
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
data/lib/good_job/scheduler.rb
CHANGED
@@ -1,11 +1,9 @@
|
|
1
|
-
require "concurrent/scheduled_task"
|
2
1
|
require "concurrent/executor/thread_pool_executor"
|
2
|
+
require "concurrent/timer_task"
|
3
3
|
require "concurrent/utility/processor_counter"
|
4
4
|
|
5
5
|
module GoodJob
|
6
6
|
class Scheduler
|
7
|
-
MAX_THREADS = Concurrent.processor_count
|
8
|
-
|
9
7
|
DEFAULT_TIMER_OPTIONS = {
|
10
8
|
execution_interval: 1,
|
11
9
|
timeout_interval: 1,
|
@@ -15,77 +13,67 @@ module GoodJob
|
|
15
13
|
DEFAULT_POOL_OPTIONS = {
|
16
14
|
name: 'good_job',
|
17
15
|
min_threads: 0,
|
18
|
-
max_threads:
|
16
|
+
max_threads: Concurrent.processor_count,
|
19
17
|
auto_terminate: true,
|
20
18
|
idletime: 0,
|
21
19
|
max_queue: 0,
|
22
20
|
fallback_policy: :abort, # shouldn't matter -- 0 max queue
|
23
21
|
}.freeze
|
24
22
|
|
25
|
-
def initialize(
|
26
|
-
|
23
|
+
def initialize(performer, timer_options: {}, pool_options: {})
|
24
|
+
raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
|
27
25
|
|
28
|
-
@
|
29
|
-
@
|
26
|
+
@performer = performer
|
27
|
+
@pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
|
28
|
+
@timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
|
30
29
|
idle_threads = @pool.max_length - @pool.length
|
31
30
|
create_thread if idle_threads.positive?
|
32
31
|
end
|
33
|
-
@timer.add_observer(
|
32
|
+
@timer.add_observer(self, :timer_observer)
|
34
33
|
@timer.execute
|
35
34
|
end
|
36
35
|
|
37
|
-
def ordered_query
|
38
|
-
@query.where("scheduled_at < ?", Time.current).or(@query.where(scheduled_at: nil)).order(priority: :desc)
|
39
|
-
end
|
40
|
-
|
41
36
|
def execute
|
42
37
|
end
|
43
38
|
|
44
39
|
def shutdown(wait: true)
|
45
|
-
|
46
|
-
@timer.shutdown
|
47
|
-
@timer.wait_for_termination if wait
|
48
|
-
end
|
40
|
+
@_shutdown = true
|
49
41
|
|
50
|
-
|
51
|
-
|
52
|
-
@
|
42
|
+
ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
|
43
|
+
ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
|
44
|
+
if @timer.running?
|
45
|
+
@timer.shutdown
|
46
|
+
@timer.wait_for_termination if wait
|
47
|
+
end
|
48
|
+
|
49
|
+
if @pool.running?
|
50
|
+
@pool.shutdown
|
51
|
+
@pool.wait_for_termination if wait
|
52
|
+
end
|
53
53
|
end
|
54
|
+
end
|
54
55
|
|
55
|
-
|
56
|
+
def shutdown?
|
57
|
+
@_shutdown
|
56
58
|
end
|
57
59
|
|
58
60
|
def create_thread
|
59
|
-
future = Concurrent::Future.new(args: [
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
break unless good_job
|
64
|
-
|
65
|
-
ActiveSupport::Notifications.instrument("job_started.good_job", { good_job: good_job })
|
66
|
-
|
67
|
-
JobWrapper.new(good_job).perform
|
68
|
-
|
69
|
-
good_job.advisory_unlock
|
70
|
-
end
|
71
|
-
end
|
72
|
-
|
73
|
-
true
|
61
|
+
future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
|
62
|
+
result = nil
|
63
|
+
Rails.application.executor.wrap { result = performer.next }
|
64
|
+
result
|
74
65
|
end
|
75
|
-
future.add_observer(
|
66
|
+
future.add_observer(self, :task_observer)
|
76
67
|
future.execute
|
77
68
|
end
|
78
69
|
|
79
|
-
|
80
|
-
|
81
|
-
ActiveSupport::Notifications.instrument("timer_task_finished.good_job", { result: result, error: error, time: time })
|
82
|
-
end
|
70
|
+
def timer_observer(time, executed_task, error)
|
71
|
+
ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
|
83
72
|
end
|
84
73
|
|
85
|
-
|
86
|
-
|
87
|
-
|
88
|
-
end
|
74
|
+
def task_observer(time, result, error)
|
75
|
+
ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
|
76
|
+
create_thread if result
|
89
77
|
end
|
90
78
|
end
|
91
79
|
end
|
data/lib/good_job/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: good_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.6.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ben Sheldon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-
|
11
|
+
date: 2020-07-15 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: concurrent-ruby
|
@@ -94,6 +94,20 @@ dependencies:
|
|
94
94
|
- - ">="
|
95
95
|
- !ruby/object:Gem::Version
|
96
96
|
version: '0'
|
97
|
+
- !ruby/object:Gem::Dependency
|
98
|
+
name: foreman
|
99
|
+
requirement: !ruby/object:Gem::Requirement
|
100
|
+
requirements:
|
101
|
+
- - ">="
|
102
|
+
- !ruby/object:Gem::Version
|
103
|
+
version: '0'
|
104
|
+
type: :development
|
105
|
+
prerelease: false
|
106
|
+
version_requirements: !ruby/object:Gem::Requirement
|
107
|
+
requirements:
|
108
|
+
- - ">="
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
97
111
|
- !ruby/object:Gem::Dependency
|
98
112
|
name: gem-release
|
99
113
|
requirement: !ruby/object:Gem::Requirement
|
@@ -122,6 +136,20 @@ dependencies:
|
|
122
136
|
- - ">="
|
123
137
|
- !ruby/object:Gem::Version
|
124
138
|
version: '0'
|
139
|
+
- !ruby/object:Gem::Dependency
|
140
|
+
name: pry
|
141
|
+
requirement: !ruby/object:Gem::Requirement
|
142
|
+
requirements:
|
143
|
+
- - ">="
|
144
|
+
- !ruby/object:Gem::Version
|
145
|
+
version: '0'
|
146
|
+
type: :development
|
147
|
+
prerelease: false
|
148
|
+
version_requirements: !ruby/object:Gem::Requirement
|
149
|
+
requirements:
|
150
|
+
- - ">="
|
151
|
+
- !ruby/object:Gem::Version
|
152
|
+
version: '0'
|
125
153
|
- !ruby/object:Gem::Dependency
|
126
154
|
name: rspec-rails
|
127
155
|
requirement: !ruby/object:Gem::Requirement
|
@@ -179,14 +207,14 @@ files:
|
|
179
207
|
- LICENSE.txt
|
180
208
|
- README.md
|
181
209
|
- exe/good_job
|
210
|
+
- lib/active_job/queue_adapters/good_job_adapter.rb
|
182
211
|
- lib/good_job.rb
|
183
212
|
- lib/good_job/adapter.rb
|
184
213
|
- lib/good_job/cli.rb
|
185
|
-
- lib/good_job/inline_scheduler.rb
|
186
214
|
- lib/good_job/job.rb
|
187
|
-
- lib/good_job/job_wrapper.rb
|
188
215
|
- lib/good_job/lockable.rb
|
189
216
|
- lib/good_job/logging.rb
|
217
|
+
- lib/good_job/pg_locks.rb
|
190
218
|
- lib/good_job/railtie.rb
|
191
219
|
- lib/good_job/scheduler.rb
|
192
220
|
- lib/good_job/version.rb
|
@@ -221,7 +249,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
221
249
|
- !ruby/object:Gem::Version
|
222
250
|
version: '0'
|
223
251
|
requirements: []
|
224
|
-
rubygems_version: 3.
|
252
|
+
rubygems_version: 3.0.3
|
225
253
|
signing_key:
|
226
254
|
specification_version: 4
|
227
255
|
summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
|
data/lib/good_job/job_wrapper.rb
DELETED
@@ -1,16 +0,0 @@
|
|
1
|
-
module GoodJob
|
2
|
-
class JobWrapper
|
3
|
-
def initialize(good_job)
|
4
|
-
@good_job = good_job
|
5
|
-
end
|
6
|
-
|
7
|
-
def perform
|
8
|
-
serialized_params = @good_job.serialized_params.merge(
|
9
|
-
"provider_job_id" => @good_job.id
|
10
|
-
)
|
11
|
-
ActiveJob::Base.execute(serialized_params)
|
12
|
-
|
13
|
-
@good_job.destroy!
|
14
|
-
end
|
15
|
-
end
|
16
|
-
end
|