good_job 3.8.0 → 3.9.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -1
- data/README.md +22 -1
- data/app/models/good_job/execution.rb +30 -22
- data/lib/good_job/active_job_extensions/concurrency.rb +50 -33
- data/lib/good_job/adapter.rb +69 -1
- data/lib/good_job/bulk.rb +120 -0
- data/lib/good_job/notifier.rb +27 -23
- data/lib/good_job/scheduler.rb +13 -0
- data/lib/good_job/version.rb +1 -1
- data/lib/good_job.rb +1 -0
- metadata +5 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 69560b4af2d1cab2a3783fb80573794066c3704aacafa7bd315a9dc6e1b75441
|
4
|
+
data.tar.gz: 55dff98e3c8ebbcebd5f09664d02f4ddd23b298c0ee2bc7cb163e43f6963de96
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: dd4763e6d473a22ea7b0f6f7b501249c7e5e29d06c1fdd9b3d028c923ba407fb485eea748fd37857a5eb09f5b26659d67f4792f4a66848542bfc1f47f7384577
|
7
|
+
data.tar.gz: 89e165d003d156fa16450762b8a8a269172509557ab59be60e7d54794d079c9963f8c11f45047ecfdea8e0963d901c8d9e1c297743392a05817d21d417c78dab
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,24 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v3.9.0](https://github.com/bensheldon/good_job/tree/v3.9.0) (2023-01-31)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.8.0...v3.9.0)
|
6
|
+
|
7
|
+
**Implemented enhancements:**
|
8
|
+
|
9
|
+
- Abort enqueue when the concurrency limit is reached [\#820](https://github.com/bensheldon/good_job/pull/820) ([TAGraves](https://github.com/TAGraves))
|
10
|
+
- Add bulk enqueue functionality [\#790](https://github.com/bensheldon/good_job/pull/790) ([julik](https://github.com/julik))
|
11
|
+
|
12
|
+
**Merged pull requests:**
|
13
|
+
|
14
|
+
- Bump alex-page/github-project-automation-plus from 0.8.2 to 0.8.3 [\#819](https://github.com/bensheldon/good_job/pull/819) ([dependabot[bot]](https://github.com/apps/dependabot))
|
15
|
+
- Bump concurrent-ruby from 1.1.10 to 1.2.0 [\#818](https://github.com/bensheldon/good_job/pull/818) ([dependabot[bot]](https://github.com/apps/dependabot))
|
16
|
+
- Bump rails from 6.1.7 to 6.1.7.2 [\#817](https://github.com/bensheldon/good_job/pull/817) ([dependabot[bot]](https://github.com/apps/dependabot))
|
17
|
+
- Bump selenium-webdriver from 4.7.1 to 4.8.0 [\#816](https://github.com/bensheldon/good_job/pull/816) ([dependabot[bot]](https://github.com/apps/dependabot))
|
18
|
+
- Bump rubocop from 1.43.0 to 1.44.1 [\#815](https://github.com/bensheldon/good_job/pull/815) ([dependabot[bot]](https://github.com/apps/dependabot))
|
19
|
+
- Ensure that anytime the Notifier uses autoloaded constants \(ActiveRecord\), they are wrapped with a Rails Executor [\#797](https://github.com/bensheldon/good_job/pull/797) ([bensheldon](https://github.com/bensheldon))
|
20
|
+
- Remove support for Ruby 2.5 and JRuby 9.2; reactivate appraisal tests for Rails HEAD [\#756](https://github.com/bensheldon/good_job/pull/756) ([bensheldon](https://github.com/bensheldon))
|
21
|
+
|
3
22
|
## [v3.8.0](https://github.com/bensheldon/good_job/tree/v3.8.0) (2023-01-27)
|
4
23
|
|
5
24
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.7.4...v3.8.0)
|
@@ -68,13 +87,16 @@
|
|
68
87
|
|
69
88
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.7.1...v3.7.2)
|
70
89
|
|
90
|
+
**Fixed bugs:**
|
91
|
+
|
92
|
+
- Ignore ActiveJob::DeserializationError when discarding jobs [\#771](https://github.com/bensheldon/good_job/pull/771) ([nickcampbell18](https://github.com/nickcampbell18))
|
93
|
+
|
71
94
|
**Closed issues:**
|
72
95
|
|
73
96
|
- Unable to discard failed jobs which crashed with `ActiveJob::DeserializationError` [\#770](https://github.com/bensheldon/good_job/issues/770)
|
74
97
|
|
75
98
|
**Merged pull requests:**
|
76
99
|
|
77
|
-
- Ignore ActiveJob::DeserializationError when discarding jobs [\#771](https://github.com/bensheldon/good_job/pull/771) ([nickcampbell18](https://github.com/nickcampbell18))
|
78
100
|
- Bump rubocop from 1.39.0 to 1.40.0 [\#769](https://github.com/bensheldon/good_job/pull/769) ([dependabot[bot]](https://github.com/apps/dependabot))
|
79
101
|
|
80
102
|
## [v3.7.1](https://github.com/bensheldon/good_job/tree/v3.7.1) (2022-12-12)
|
data/README.md
CHANGED
@@ -58,6 +58,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
58
58
|
- [Database connections](#database-connections)
|
59
59
|
- [Production setup](#production-setup)
|
60
60
|
- [Queue performance with Queue Select Limit](#queue-performance-with-queue-select-limit)
|
61
|
+
- [Bulk enqueue](#bulk-enqueue)
|
61
62
|
- [Execute jobs async / in-process](#execute-jobs-async--in-process)
|
62
63
|
- [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
|
63
64
|
- [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
|
@@ -146,7 +147,7 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
146
147
|
## Compatibility
|
147
148
|
|
148
149
|
- **Ruby on Rails:** 6.0+
|
149
|
-
- **Ruby:** Ruby 2.
|
150
|
+
- **Ruby:** Ruby 2.6+. JRuby 9.3+
|
150
151
|
- **Postgres:** 10.0+
|
151
152
|
|
152
153
|
## Configuration
|
@@ -793,6 +794,26 @@ To explain where this value is used, here is the pseudo-query that GoodJob uses
|
|
793
794
|
)
|
794
795
|
```
|
795
796
|
|
797
|
+
### Bulk enqueue
|
798
|
+
|
799
|
+
GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
|
800
|
+
|
801
|
+
```ruby
|
802
|
+
# Capture jobs using `.perform_later`:
|
803
|
+
active_jobs = GoodJob::Bulk.enqueue do
|
804
|
+
MyJob.perform_later
|
805
|
+
AnotherJob.perform_later
|
806
|
+
# If an exception is raised within this block, no jobs will be inserted.
|
807
|
+
end
|
808
|
+
|
809
|
+
# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
|
810
|
+
# Jobs that have been successfully enqueued have a `provider_job_id` set.
|
811
|
+
active_jobs.all?(&:provider_job_id)
|
812
|
+
|
813
|
+
# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
|
814
|
+
GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
|
815
|
+
```
|
816
|
+
|
796
817
|
### Execute jobs async / in-process
|
797
818
|
|
798
819
|
GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
|
@@ -43,10 +43,10 @@ module GoodJob
|
|
43
43
|
case string.first
|
44
44
|
when '-'
|
45
45
|
exclude_queues = true
|
46
|
-
string = string[1
|
46
|
+
string = string[1..]
|
47
47
|
when '+'
|
48
48
|
ordered_queues = true
|
49
|
-
string = string[1
|
49
|
+
string = string[1..]
|
50
50
|
end
|
51
51
|
|
52
52
|
queues = string.split(',').map(&:strip)
|
@@ -197,6 +197,28 @@ module GoodJob
|
|
197
197
|
end
|
198
198
|
end)
|
199
199
|
|
200
|
+
# Construct a GoodJob::Execution from an ActiveJob instance.
|
201
|
+
def self.build_for_enqueue(active_job, overrides = {})
|
202
|
+
execution_args = {
|
203
|
+
active_job_id: active_job.job_id,
|
204
|
+
queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
|
205
|
+
priority: active_job.priority || DEFAULT_PRIORITY,
|
206
|
+
serialized_params: active_job.serialize,
|
207
|
+
scheduled_at: active_job.scheduled_at,
|
208
|
+
}
|
209
|
+
|
210
|
+
execution_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
211
|
+
|
212
|
+
if CurrentThread.cron_key
|
213
|
+
execution_args[:cron_key] = CurrentThread.cron_key
|
214
|
+
execution_args[:cron_at] = CurrentThread.cron_at
|
215
|
+
elsif CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
216
|
+
execution_args[:cron_key] = CurrentThread.execution.cron_key
|
217
|
+
end
|
218
|
+
|
219
|
+
new(**execution_args.merge(overrides))
|
220
|
+
end
|
221
|
+
|
200
222
|
# Finds the next eligible Execution, acquire an advisory lock related to it, and
|
201
223
|
# executes the job.
|
202
224
|
# @return [ExecutionResult, nil]
|
@@ -244,33 +266,19 @@ module GoodJob
|
|
244
266
|
# @param active_job [ActiveJob::Base]
|
245
267
|
# The job to enqueue.
|
246
268
|
# @param scheduled_at [Float]
|
247
|
-
# Epoch timestamp when the job should be executed
|
269
|
+
# Epoch timestamp when the job should be executed, if blank will delegate to the ActiveJob instance
|
248
270
|
# @param create_with_advisory_lock [Boolean]
|
249
271
|
# Whether to establish a lock on the {Execution} record after it is created.
|
272
|
+
# @param persist_immediately [Boolean]
|
273
|
+
# Whether to save the record immediately or just initialize it with values. When bulk-inserting
|
274
|
+
# jobs the caller takes care of the persistence and sets this parameter to `false`
|
250
275
|
# @return [Execution]
|
251
276
|
# The new {Execution} instance representing the queued ActiveJob job.
|
252
277
|
def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
|
253
278
|
ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
|
254
|
-
|
255
|
-
active_job_id: active_job.job_id,
|
256
|
-
queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
|
257
|
-
priority: active_job.priority || DEFAULT_PRIORITY,
|
258
|
-
serialized_params: active_job.serialize,
|
259
|
-
scheduled_at: scheduled_at,
|
260
|
-
create_with_advisory_lock: create_with_advisory_lock,
|
261
|
-
}
|
262
|
-
|
263
|
-
execution_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
264
|
-
|
265
|
-
if CurrentThread.cron_key
|
266
|
-
execution_args[:cron_key] = CurrentThread.cron_key
|
267
|
-
execution_args[:cron_at] = CurrentThread.cron_at
|
268
|
-
elsif CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
269
|
-
execution_args[:cron_key] = CurrentThread.execution.cron_key
|
270
|
-
end
|
271
|
-
|
272
|
-
execution = GoodJob::Execution.new(**execution_args)
|
279
|
+
execution = build_for_enqueue(active_job, { scheduled_at: scheduled_at })
|
273
280
|
|
281
|
+
execution.create_with_advisory_lock = create_with_advisory_lock
|
274
282
|
instrument_payload[:execution] = execution
|
275
283
|
|
276
284
|
execution.save!
|
@@ -25,40 +25,13 @@ module GoodJob
|
|
25
25
|
class_attribute :good_job_concurrency_config, instance_accessor: false, default: {}
|
26
26
|
attr_writer :good_job_concurrency_key
|
27
27
|
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
# Always allow jobs to be retried because the current job's execution will complete momentarily
|
33
|
-
next(block.call) if CurrentThread.active_job_id == job.job_id
|
34
|
-
|
35
|
-
# Only generate the concurrency key on the initial enqueue in case it is dynamic
|
36
|
-
job.good_job_concurrency_key ||= job._good_job_concurrency_key
|
37
|
-
key = job.good_job_concurrency_key
|
38
|
-
next(block.call) if key.blank?
|
39
|
-
|
40
|
-
enqueue_limit = job.class.good_job_concurrency_config[:enqueue_limit]
|
41
|
-
enqueue_limit = instance_exec(&enqueue_limit) if enqueue_limit.respond_to?(:call)
|
42
|
-
enqueue_limit = nil unless enqueue_limit.present? && (0...Float::INFINITY).cover?(enqueue_limit)
|
43
|
-
|
44
|
-
unless enqueue_limit
|
45
|
-
total_limit = job.class.good_job_concurrency_config[:total_limit]
|
46
|
-
total_limit = instance_exec(&total_limit) if total_limit.respond_to?(:call)
|
47
|
-
total_limit = nil unless total_limit.present? && (0...Float::INFINITY).cover?(total_limit)
|
28
|
+
if ActiveJob.gem_version >= Gem::Version.new("6.1.0")
|
29
|
+
before_enqueue do |job|
|
30
|
+
good_job_enqueue_concurrency_check(job, on_abort: -> { throw(:abort) }, on_enqueue: nil)
|
48
31
|
end
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
GoodJob::Execution.advisory_lock_key(key, function: "pg_advisory_lock") do
|
54
|
-
enqueue_concurrency = if enqueue_limit
|
55
|
-
GoodJob::Execution.where(concurrency_key: key).unfinished.advisory_unlocked.count
|
56
|
-
else
|
57
|
-
GoodJob::Execution.where(concurrency_key: key).unfinished.count
|
58
|
-
end
|
59
|
-
|
60
|
-
# The job has not yet been enqueued, so check if adding it will go over the limit
|
61
|
-
block.call unless (enqueue_concurrency + 1) > limit
|
32
|
+
else
|
33
|
+
around_enqueue do |job, block|
|
34
|
+
good_job_enqueue_concurrency_check(job, on_abort: nil, on_enqueue: block)
|
62
35
|
end
|
63
36
|
end
|
64
37
|
|
@@ -113,6 +86,50 @@ module GoodJob
|
|
113
86
|
@good_job_concurrency_key || _good_job_concurrency_key
|
114
87
|
end
|
115
88
|
|
89
|
+
private
|
90
|
+
|
91
|
+
def good_job_enqueue_concurrency_check(job, on_abort:, on_enqueue:)
|
92
|
+
# Don't attempt to enforce concurrency limits with other queue adapters.
|
93
|
+
return on_enqueue&.call unless job.class.queue_adapter.is_a?(GoodJob::Adapter)
|
94
|
+
|
95
|
+
# Always allow jobs to be retried because the current job's execution will complete momentarily
|
96
|
+
return on_enqueue&.call if CurrentThread.active_job_id == job.job_id
|
97
|
+
|
98
|
+
# Only generate the concurrency key on the initial enqueue in case it is dynamic
|
99
|
+
job.good_job_concurrency_key ||= job._good_job_concurrency_key
|
100
|
+
key = job.good_job_concurrency_key
|
101
|
+
return on_enqueue&.call if key.blank?
|
102
|
+
|
103
|
+
enqueue_limit = job.class.good_job_concurrency_config[:enqueue_limit]
|
104
|
+
enqueue_limit = instance_exec(&enqueue_limit) if enqueue_limit.respond_to?(:call)
|
105
|
+
enqueue_limit = nil unless enqueue_limit.present? && (0...Float::INFINITY).cover?(enqueue_limit)
|
106
|
+
|
107
|
+
unless enqueue_limit
|
108
|
+
total_limit = job.class.good_job_concurrency_config[:total_limit]
|
109
|
+
total_limit = instance_exec(&total_limit) if total_limit.respond_to?(:call)
|
110
|
+
total_limit = nil unless total_limit.present? && (0...Float::INFINITY).cover?(total_limit)
|
111
|
+
end
|
112
|
+
|
113
|
+
limit = enqueue_limit || total_limit
|
114
|
+
return on_enqueue&.call unless limit
|
115
|
+
|
116
|
+
GoodJob::Execution.advisory_lock_key(key, function: "pg_advisory_lock") do
|
117
|
+
enqueue_concurrency = if enqueue_limit
|
118
|
+
GoodJob::Execution.where(concurrency_key: key).unfinished.advisory_unlocked.count
|
119
|
+
else
|
120
|
+
GoodJob::Execution.where(concurrency_key: key).unfinished.count
|
121
|
+
end
|
122
|
+
|
123
|
+
# The job has not yet been enqueued, so check if adding it will go over the limit
|
124
|
+
if (enqueue_concurrency + 1) > limit
|
125
|
+
logger.info "Aborted enqueue of #{job.class.name} (Job ID: #{job.job_id}) because the concurrency key '#{key}' has reached its limit of #{limit} #{'job'.pluralize(limit)}"
|
126
|
+
on_abort&.call
|
127
|
+
else
|
128
|
+
on_enqueue&.call
|
129
|
+
end
|
130
|
+
end
|
131
|
+
end
|
132
|
+
|
116
133
|
# Generates the concurrency key from the configuration
|
117
134
|
# @return [Object] concurrency key
|
118
135
|
def _good_job_concurrency_key
|
data/lib/good_job/adapter.rb
CHANGED
@@ -40,6 +40,70 @@ module GoodJob
|
|
40
40
|
enqueue_at(active_job, nil)
|
41
41
|
end
|
42
42
|
|
43
|
+
# Enqueues multiple ActiveJob instances at once
|
44
|
+
# @param active_jobs [Array<ActiveJob::Base>] jobs to be enqueued
|
45
|
+
# @return [Integer] number of jobs that were successfully enqueued
|
46
|
+
def enqueue_all(active_jobs)
|
47
|
+
active_jobs = Array(active_jobs)
|
48
|
+
return 0 if active_jobs.empty?
|
49
|
+
|
50
|
+
current_time = Time.current
|
51
|
+
executions = active_jobs.map do |active_job|
|
52
|
+
GoodJob::Execution.build_for_enqueue(active_job, {
|
53
|
+
id: SecureRandom.uuid,
|
54
|
+
created_at: current_time,
|
55
|
+
updated_at: current_time,
|
56
|
+
})
|
57
|
+
end
|
58
|
+
|
59
|
+
inline_executions = []
|
60
|
+
GoodJob::Execution.transaction(requires_new: true, joinable: false) do
|
61
|
+
results = GoodJob::Execution.insert_all(executions.map(&:attributes), returning: %w[id active_job_id]) # rubocop:disable Rails/SkipsModelValidations
|
62
|
+
|
63
|
+
job_id_to_provider_job_id = results.each_with_object({}) { |result, hash| hash[result['active_job_id']] = result['id'] }
|
64
|
+
active_jobs.each do |active_job|
|
65
|
+
active_job.provider_job_id = job_id_to_provider_job_id[active_job.job_id]
|
66
|
+
end
|
67
|
+
executions.each do |execution|
|
68
|
+
execution.instance_variable_set(:@new_record, false) if job_id_to_provider_job_id[execution.active_job_id]
|
69
|
+
end
|
70
|
+
executions = executions.select(&:persisted?) # prune unpersisted executions
|
71
|
+
|
72
|
+
if execute_inline?
|
73
|
+
inline_executions = executions.select { |execution| (execution.scheduled_at.nil? || execution.scheduled_at <= Time.current) }
|
74
|
+
inline_executions.each(&:advisory_lock!)
|
75
|
+
end
|
76
|
+
end
|
77
|
+
|
78
|
+
begin
|
79
|
+
until inline_executions.empty?
|
80
|
+
begin
|
81
|
+
inline_execution = inline_executions.shift
|
82
|
+
inline_result = inline_execution.perform
|
83
|
+
ensure
|
84
|
+
inline_execution.advisory_unlock
|
85
|
+
inline_execution.run_callbacks(:perform_unlocked)
|
86
|
+
end
|
87
|
+
raise inline_result.unhandled_error if inline_result.unhandled_error
|
88
|
+
end
|
89
|
+
ensure
|
90
|
+
inline_executions.each(&:advisory_unlock)
|
91
|
+
end
|
92
|
+
|
93
|
+
executions.reject(&:finished_at).group_by(&:queue_name).each do |queue_name, executions_by_queue|
|
94
|
+
executions_by_queue.group_by(&:scheduled_at).each do |scheduled_at, executions_by_queue_and_scheduled_at|
|
95
|
+
# TODO: have Adapter#create_thread handle state[:count] values
|
96
|
+
state = { queue_name: queue_name, count: executions_by_queue_and_scheduled_at.size }
|
97
|
+
state[:scheduled_at] = scheduled_at if scheduled_at
|
98
|
+
|
99
|
+
executed_locally = execute_async? && @scheduler&.create_thread(state)
|
100
|
+
Notifier.notify(state) unless executed_locally
|
101
|
+
end
|
102
|
+
end
|
103
|
+
|
104
|
+
active_jobs.count(&:provider_job_id)
|
105
|
+
end
|
106
|
+
|
43
107
|
# Enqueues an ActiveJob job to be run at a specific time.
|
44
108
|
# For use by Rails; you should generally not call this directly.
|
45
109
|
# @param active_job [ActiveJob::Base] the job to be enqueued from +#perform_later+
|
@@ -47,8 +111,12 @@ module GoodJob
|
|
47
111
|
# @return [GoodJob::Execution]
|
48
112
|
def enqueue_at(active_job, timestamp)
|
49
113
|
scheduled_at = timestamp ? Time.zone.at(timestamp) : nil
|
50
|
-
will_execute_inline = execute_inline? && (scheduled_at.nil? || scheduled_at <= Time.current)
|
51
114
|
|
115
|
+
# If there is a currently open Bulk in the current thread, direct the
|
116
|
+
# job there to be enqueued using enqueue_all
|
117
|
+
return if GoodJob::Bulk.capture(active_job, queue_adapter: self)
|
118
|
+
|
119
|
+
will_execute_inline = execute_inline? && (scheduled_at.nil? || scheduled_at <= Time.current)
|
52
120
|
execution = GoodJob::Execution.enqueue(
|
53
121
|
active_job,
|
54
122
|
scheduled_at: scheduled_at,
|
@@ -0,0 +1,120 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
require 'active_support/core_ext/module/attribute_accessors_per_thread'
|
3
|
+
|
4
|
+
module GoodJob
|
5
|
+
module Bulk
|
6
|
+
Error = Class.new(StandardError)
|
7
|
+
|
8
|
+
# @!attribute [rw] current_buffer
|
9
|
+
# @!scope class
|
10
|
+
# Current buffer of jobs to be enqueued.
|
11
|
+
# @return [GoodJob::Bulk::Buffer, nil]
|
12
|
+
thread_mattr_accessor :current_buffer
|
13
|
+
|
14
|
+
# Capture jobs to a buffer. Pass either a block, or specific Active Jobs to be buffered.
|
15
|
+
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be buffered.
|
16
|
+
# @param queue_adapter Override the jobs implict queue adapter with an explicit one.
|
17
|
+
# @return [nil, Array<ActiveJob::Base>] The ActiveJob instances that have been buffered; nil if no active buffer
|
18
|
+
def self.capture(active_jobs = nil, queue_adapter: nil)
|
19
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block_given? && active_jobs
|
20
|
+
|
21
|
+
if block_given?
|
22
|
+
begin
|
23
|
+
original_buffer = current_buffer
|
24
|
+
self.current_buffer = Buffer.new
|
25
|
+
yield
|
26
|
+
current_buffer.active_jobs
|
27
|
+
ensure
|
28
|
+
self.current_buffer = original_buffer
|
29
|
+
end
|
30
|
+
else
|
31
|
+
current_buffer&.add(active_jobs, queue_adapter: queue_adapter)
|
32
|
+
end
|
33
|
+
end
|
34
|
+
|
35
|
+
# Capture jobs to a buffer and enqueue them all at once; or enqueue the current buffer.
|
36
|
+
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be enqueued.
|
37
|
+
# @return [Array<ActiveJob::Base>] The ActiveJob instances that have been captured; check provider_job_id to confirm enqueued.
|
38
|
+
def self.enqueue(active_jobs = nil)
|
39
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block_given? && active_jobs
|
40
|
+
|
41
|
+
if block_given?
|
42
|
+
capture do
|
43
|
+
yield
|
44
|
+
current_buffer&.enqueue
|
45
|
+
end
|
46
|
+
elsif active_jobs.present?
|
47
|
+
buffer = Buffer.new
|
48
|
+
buffer.add(active_jobs)
|
49
|
+
buffer.enqueue
|
50
|
+
buffer.active_jobs
|
51
|
+
elsif current_buffer.present?
|
52
|
+
current_buffer.enqueue
|
53
|
+
current_buffer.active_jobs
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
# Temporarily unset the current buffer; used to enqueue buffered jobs.
|
58
|
+
# @return [void]
|
59
|
+
def self.unbuffer
|
60
|
+
original_buffer = current_buffer
|
61
|
+
self.current_buffer = nil
|
62
|
+
yield
|
63
|
+
ensure
|
64
|
+
self.current_buffer = original_buffer
|
65
|
+
end
|
66
|
+
|
67
|
+
class Buffer
|
68
|
+
def initialize
|
69
|
+
@values = []
|
70
|
+
end
|
71
|
+
|
72
|
+
def add(active_jobs, queue_adapter: nil)
|
73
|
+
new_pairs = Array(active_jobs).map do |active_job|
|
74
|
+
adapter = queue_adapter || active_job.class.queue_adapter
|
75
|
+
raise Error, "Jobs must have a Queue Adapter" unless adapter
|
76
|
+
|
77
|
+
[active_job, adapter]
|
78
|
+
end
|
79
|
+
@values.append(*new_pairs)
|
80
|
+
|
81
|
+
true
|
82
|
+
end
|
83
|
+
|
84
|
+
def enqueue
|
85
|
+
Bulk.unbuffer do
|
86
|
+
active_jobs_by_queue_adapter.each do |adapter, jobs|
|
87
|
+
jobs = jobs.reject(&:provider_job_id) # Do not re-enqueue already enqueued jobs
|
88
|
+
|
89
|
+
if adapter.respond_to?(:enqueue_all)
|
90
|
+
unbulkable_jobs, bulkable_jobs = jobs.partition do |job|
|
91
|
+
job.respond_to?(:good_job_concurrency_key) && job.good_job_concurrency_key &&
|
92
|
+
(job.class.good_job_concurrency_config[:enqueue_limit] || job.class.good_job_concurrency_config[:total_limit])
|
93
|
+
end
|
94
|
+
adapter.enqueue_all(bulkable_jobs) if bulkable_jobs.any?
|
95
|
+
else
|
96
|
+
unbulkable_jobs = jobs
|
97
|
+
end
|
98
|
+
|
99
|
+
unbulkable_jobs.each do |job|
|
100
|
+
job.enqueue
|
101
|
+
rescue GoodJob::ActiveJobExtensions::Concurrency::ConcurrencyExceededError
|
102
|
+
# ignore
|
103
|
+
end
|
104
|
+
end
|
105
|
+
end
|
106
|
+
end
|
107
|
+
|
108
|
+
def active_jobs_by_queue_adapter
|
109
|
+
@values.each_with_object({}) do |(job, adapter), memo|
|
110
|
+
memo[adapter] ||= []
|
111
|
+
memo[adapter] << job
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
115
|
+
def active_jobs
|
116
|
+
@values.map(&:first)
|
117
|
+
end
|
118
|
+
end
|
119
|
+
end
|
120
|
+
end
|
data/lib/good_job/notifier.rb
CHANGED
@@ -168,35 +168,37 @@ module GoodJob # :nodoc:
|
|
168
168
|
future = Concurrent::ScheduledTask.new(delay, args: [@recipients, executor, @listening], executor: @executor) do |thr_recipients, thr_executor, thr_listening|
|
169
169
|
with_connection do
|
170
170
|
begin
|
171
|
-
|
172
|
-
|
173
|
-
|
171
|
+
Rails.application.executor.wrap do
|
172
|
+
run_callbacks :listen do
|
173
|
+
ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
|
174
|
+
connection.execute("LISTEN #{CHANNEL}")
|
175
|
+
end
|
176
|
+
thr_listening.make_true
|
174
177
|
end
|
175
|
-
thr_listening.make_true
|
176
178
|
end
|
177
179
|
|
178
|
-
|
179
|
-
|
180
|
-
|
181
|
-
next unless channel == CHANNEL
|
182
|
-
|
183
|
-
ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
|
184
|
-
parsed_payload = JSON.parse(payload, symbolize_names: true)
|
185
|
-
thr_recipients.each do |recipient|
|
186
|
-
target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
|
187
|
-
target.send(method_name, parsed_payload)
|
188
|
-
end
|
189
|
-
end
|
180
|
+
while thr_executor.running?
|
181
|
+
wait_for_notify do |channel, payload|
|
182
|
+
next unless channel == CHANNEL
|
190
183
|
|
191
|
-
|
184
|
+
ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
|
185
|
+
parsed_payload = JSON.parse(payload, symbolize_names: true)
|
186
|
+
thr_recipients.each do |recipient|
|
187
|
+
target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
|
188
|
+
target.send(method_name, parsed_payload)
|
189
|
+
end
|
192
190
|
end
|
191
|
+
|
192
|
+
reset_connection_errors
|
193
193
|
end
|
194
194
|
end
|
195
195
|
ensure
|
196
|
-
|
197
|
-
|
198
|
-
|
199
|
-
|
196
|
+
Rails.application.executor.wrap do
|
197
|
+
run_callbacks :unlisten do
|
198
|
+
thr_listening.make_false
|
199
|
+
ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
|
200
|
+
connection.execute("UNLISTEN *")
|
201
|
+
end
|
200
202
|
end
|
201
203
|
end
|
202
204
|
end
|
@@ -207,8 +209,10 @@ module GoodJob # :nodoc:
|
|
207
209
|
end
|
208
210
|
|
209
211
|
def with_connection
|
210
|
-
|
211
|
-
Execution.connection_pool.
|
212
|
+
Rails.application.executor.wrap do
|
213
|
+
self.connection = Execution.connection_pool.checkout.tap do |conn|
|
214
|
+
Execution.connection_pool.remove(conn)
|
215
|
+
end
|
212
216
|
end
|
213
217
|
connection.execute("SET application_name = #{connection.quote(self.class.name)}")
|
214
218
|
|
data/lib/good_job/scheduler.rb
CHANGED
@@ -155,6 +155,19 @@ module GoodJob # :nodoc:
|
|
155
155
|
if state
|
156
156
|
return false unless performer.next?(state)
|
157
157
|
|
158
|
+
if state[:count]
|
159
|
+
# When given state for multiple jobs, try to create a thread for each one.
|
160
|
+
# Return true if a thread can be created for all of them, nil if partial or none.
|
161
|
+
|
162
|
+
state_without_count = state.without(:count)
|
163
|
+
result = state[:count].times do
|
164
|
+
value = create_thread(state_without_count)
|
165
|
+
break(value) unless value
|
166
|
+
end
|
167
|
+
|
168
|
+
return result.nil? ? nil : true
|
169
|
+
end
|
170
|
+
|
158
171
|
if state[:scheduled_at]
|
159
172
|
scheduled_at = if state[:scheduled_at].is_a? String
|
160
173
|
Time.zone.parse state[:scheduled_at]
|
data/lib/good_job/version.rb
CHANGED
data/lib/good_job.rb
CHANGED
@@ -10,6 +10,7 @@ require "active_job/queue_adapters/good_job_adapter"
|
|
10
10
|
require "good_job/active_job_extensions/concurrency"
|
11
11
|
|
12
12
|
require "good_job/assignable_connection"
|
13
|
+
require "good_job/bulk"
|
13
14
|
require "good_job/cleanup_tracker"
|
14
15
|
require "good_job/cli"
|
15
16
|
require "good_job/configuration"
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: good_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.
|
4
|
+
version: 3.9.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ben Sheldon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-01-
|
11
|
+
date: 2023-01-31 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -136,20 +136,6 @@ dependencies:
|
|
136
136
|
- - ">="
|
137
137
|
- !ruby/object:Gem::Version
|
138
138
|
version: '0'
|
139
|
-
- !ruby/object:Gem::Dependency
|
140
|
-
name: database_cleaner
|
141
|
-
requirement: !ruby/object:Gem::Requirement
|
142
|
-
requirements:
|
143
|
-
- - ">="
|
144
|
-
- !ruby/object:Gem::Version
|
145
|
-
version: '0'
|
146
|
-
type: :development
|
147
|
-
prerelease: false
|
148
|
-
version_requirements: !ruby/object:Gem::Requirement
|
149
|
-
requirements:
|
150
|
-
- - ">="
|
151
|
-
- !ruby/object:Gem::Version
|
152
|
-
version: '0'
|
153
139
|
- !ruby/object:Gem::Dependency
|
154
140
|
name: dotenv
|
155
141
|
requirement: !ruby/object:Gem::Requirement
|
@@ -409,6 +395,7 @@ files:
|
|
409
395
|
- lib/good_job/active_job_extensions/concurrency.rb
|
410
396
|
- lib/good_job/adapter.rb
|
411
397
|
- lib/good_job/assignable_connection.rb
|
398
|
+
- lib/good_job/bulk.rb
|
412
399
|
- lib/good_job/cleanup_tracker.rb
|
413
400
|
- lib/good_job/cli.rb
|
414
401
|
- lib/good_job/configuration.rb
|
@@ -451,14 +438,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
451
438
|
requirements:
|
452
439
|
- - ">="
|
453
440
|
- !ruby/object:Gem::Version
|
454
|
-
version: 2.
|
441
|
+
version: 2.6.0
|
455
442
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
456
443
|
requirements:
|
457
444
|
- - ">="
|
458
445
|
- !ruby/object:Gem::Version
|
459
446
|
version: '0'
|
460
447
|
requirements: []
|
461
|
-
rubygems_version: 3.
|
448
|
+
rubygems_version: 3.4.5
|
462
449
|
signing_key:
|
463
450
|
specification_version: 4
|
464
451
|
summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
|