que 1.3.0 → 2.0.0.beta1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.github/workflows/tests.yml +16 -8
- data/.ruby-version +1 -0
- data/CHANGELOG.md +415 -527
- data/Dockerfile +1 -1
- data/README.md +8 -8
- data/bin/command_line_interface.rb +4 -5
- data/docker-compose.yml +3 -2
- data/docs/README.md +9 -10
- data/lib/que/active_job/extensions.rb +46 -29
- data/lib/que/active_record/model.rb +2 -2
- data/lib/que/job.rb +14 -32
- data/lib/que/job_buffer.rb +1 -10
- data/lib/que/job_methods.rb +6 -2
- data/lib/que/locker.rb +0 -4
- data/lib/que/migrations/6/down.sql +8 -0
- data/lib/que/migrations/6/up.sql +8 -0
- data/lib/que/migrations.rb +1 -1
- data/lib/que/poller.rb +8 -3
- data/lib/que/sequel/model.rb +5 -2
- data/lib/que/utils/ruby2_keywords.rb +19 -0
- data/lib/que/version.rb +2 -2
- data/lib/que/worker.rb +1 -1
- data/lib/que.rb +4 -1
- data/que.gemspec +2 -0
- metadata +10 -6
data/Dockerfile
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
FROM ruby:
|
1
|
+
FROM ruby:3.1.1-slim-buster@sha256:2ada3e4fe7b1703c9333ad4eb9fc12c1d4d60bce0f981281b2151057e928d9ad AS base
|
2
2
|
|
3
3
|
# Install libpq-dev in our base layer, as it's needed in all environments
|
4
4
|
RUN apt-get update \
|
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# Que ![tests](https://github.com/que-rb/que/workflows/tests/badge.svg)
|
2
2
|
|
3
|
-
**This README and the rest of the docs on the master branch all refer to Que
|
3
|
+
**This README and the rest of the docs on the master branch all refer to Que 2.x. For older versions, please refer to the docs on the respective branches: [1.x](https://github.com/que-rb/que/tree/1.x), or [0.x](https://github.com/que-rb/que/tree/0.x).**
|
4
4
|
|
5
5
|
*TL;DR: Que is a high-performance job queue that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.*
|
6
6
|
|
@@ -23,9 +23,9 @@ Que's secondary goal is performance. The worker process is multithreaded, so tha
|
|
23
23
|
|
24
24
|
Compatibility:
|
25
25
|
|
26
|
-
- MRI Ruby 2.
|
26
|
+
- MRI Ruby 2.7+
|
27
27
|
- PostgreSQL 9.5+
|
28
|
-
- Rails
|
28
|
+
- Rails 6.0+ (optional)
|
29
29
|
|
30
30
|
**Please note** - Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.
|
31
31
|
|
@@ -54,12 +54,12 @@ gem install que
|
|
54
54
|
First, create the queue schema in a migration. For example:
|
55
55
|
|
56
56
|
```ruby
|
57
|
-
class CreateQueSchema < ActiveRecord::Migration[
|
57
|
+
class CreateQueSchema < ActiveRecord::Migration[6.0]
|
58
58
|
def up
|
59
59
|
# Whenever you use Que in a migration, always specify the version you're
|
60
60
|
# migrating to. If you're unsure what the current version is, check the
|
61
61
|
# changelog.
|
62
|
-
Que.migrate!(version:
|
62
|
+
Que.migrate!(version: 6)
|
63
63
|
end
|
64
64
|
|
65
65
|
def down
|
@@ -117,10 +117,10 @@ end
|
|
117
117
|
You can also add options to run the job after a specific time, or with a specific priority:
|
118
118
|
|
119
119
|
```ruby
|
120
|
-
ChargeCreditCard.enqueue
|
120
|
+
ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 })
|
121
121
|
```
|
122
122
|
## Running the Que Worker
|
123
|
-
In order to process jobs, you must start a separate worker process outside of your main server.
|
123
|
+
In order to process jobs, you must start a separate worker process outside of your main server.
|
124
124
|
|
125
125
|
```bash
|
126
126
|
bundle exec que
|
@@ -142,7 +142,7 @@ You may need to pass que a file path to require so that it can load your app. Qu
|
|
142
142
|
|
143
143
|
If you're using ActiveRecord to dump your database's schema, please [set your schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly. This is a good idea regardless, as the `:ruby` schema format doesn't support many of PostgreSQL's advanced features.
|
144
144
|
|
145
|
-
Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails.
|
145
|
+
Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. As of 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default. You can either:
|
146
146
|
|
147
147
|
- [Configure Rails](https://guides.rubyonrails.org/configuring.html) to send all internal job types to the 'default' queue by adding the following to `config/application.rb`:
|
148
148
|
|
@@ -138,10 +138,9 @@ module Que
|
|
138
138
|
opts.on(
|
139
139
|
'--minimum-buffer-size [SIZE]',
|
140
140
|
Integer,
|
141
|
-
"
|
142
|
-
"process awaiting a worker (default: 2)",
|
141
|
+
"Unused (deprecated)",
|
143
142
|
) do |s|
|
144
|
-
|
143
|
+
warn "The --minimum-buffer-size SIZE option has been deprecated and will be removed in v2.0 (it's actually been unused since v1.0.0.beta4)"
|
145
144
|
end
|
146
145
|
|
147
146
|
opts.on(
|
@@ -221,7 +220,7 @@ OUTPUT
|
|
221
220
|
|
222
221
|
locker =
|
223
222
|
begin
|
224
|
-
Que::Locker.new(options)
|
223
|
+
Que::Locker.new(**options)
|
225
224
|
rescue => e
|
226
225
|
output.puts(e.message)
|
227
226
|
return 1
|
@@ -236,7 +235,7 @@ OUTPUT
|
|
236
235
|
<<~STARTUP
|
237
236
|
Que #{Que::VERSION} started worker process with:
|
238
237
|
Worker threads: #{locker.workers.length} (priorities: #{locker.workers.map { |w| w.priority || 'any' }.join(', ')})
|
239
|
-
Buffer size: #{locker.job_buffer.
|
238
|
+
Buffer size: #{locker.job_buffer.maximum_size}
|
240
239
|
Queues:
|
241
240
|
#{locker.queues.map { |queue, interval| " - #{queue} (poll interval: #{interval}s)" }.join("\n")}
|
242
241
|
Que waiting for jobs...
|
data/docker-compose.yml
CHANGED
@@ -10,13 +10,14 @@ services:
|
|
10
10
|
- db
|
11
11
|
volumes:
|
12
12
|
- .:/work
|
13
|
-
- ruby-
|
13
|
+
- ruby-3.1.1-gem-cache:/usr/local/bundle
|
14
14
|
- ~/.docker-rc.d/:/.docker-rc.d/:ro
|
15
15
|
working_dir: /work
|
16
16
|
entrypoint: /work/scripts/docker-entrypoint
|
17
17
|
command: bash
|
18
18
|
environment:
|
19
19
|
DATABASE_URL: postgres://que:que@db/que-test
|
20
|
+
USE_RAILS: ~
|
20
21
|
|
21
22
|
db:
|
22
23
|
image: "postgres:${POSTGRES_VERSION-13}"
|
@@ -43,4 +44,4 @@ services:
|
|
43
44
|
|
44
45
|
volumes:
|
45
46
|
db-data: ~
|
46
|
-
ruby-
|
47
|
+
ruby-3.1.1-gem-cache: ~
|
data/docs/README.md
CHANGED
@@ -4,7 +4,7 @@ Docs Index
|
|
4
4
|
- [Command Line Interface](#command-line-interface)
|
5
5
|
* [worker-priorities and worker-count](#worker-priorities-and-worker-count)
|
6
6
|
* [poll-interval](#poll-interval)
|
7
|
-
* [
|
7
|
+
* [maximum-buffer-size](#maximum-buffer-size)
|
8
8
|
* [connection-url](#connection-url)
|
9
9
|
* [wait-period](#wait-period)
|
10
10
|
* [log-internals](#log-internals)
|
@@ -62,7 +62,6 @@ usage: que [options] [file/to/require] ...
|
|
62
62
|
--connection-url [URL] Set a custom database url to connect to for locking purposes.
|
63
63
|
--log-internals Log verbosely about Que's internal state. Only recommended for debugging issues
|
64
64
|
--maximum-buffer-size [SIZE] Set maximum number of jobs to be locked and held in this process awaiting a worker (default: 8)
|
65
|
-
--minimum-buffer-size [SIZE] Set minimum number of jobs to be locked and held in this process awaiting a worker (default: 2)
|
66
65
|
--wait-period [PERIOD] Set maximum interval between checks of the in-memory job queue, in milliseconds (default: 50)
|
67
66
|
```
|
68
67
|
|
@@ -82,9 +81,9 @@ If you pass both worker-count and worker-priorities, the count will trim or pad
|
|
82
81
|
|
83
82
|
This option sets the number of seconds the process will wait between polls of the job queue. Jobs that are ready to be worked immediately will be broadcast via the LISTEN/NOTIFY system, so polling is unnecessary for them - polling is only necessary for jobs that are scheduled in the future or which are being delayed due to errors. The default is 5 seconds.
|
84
83
|
|
85
|
-
###
|
84
|
+
### maximum-buffer-size
|
86
85
|
|
87
|
-
|
86
|
+
This option sets the size of the internal buffer that Que uses to hold jobs until they're ready for workers. The default maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked. If you don't want jobs to be buffered at all, you can set this value to zero.
|
88
87
|
|
89
88
|
### connection-url
|
90
89
|
|
@@ -129,8 +128,8 @@ There are other docs to read if you're using [Sequel](#using-sequel) or [plain P
|
|
129
128
|
After you've connected Que to the database, you can manage the jobs table. You'll want to migrate to a specific version in a migration file, to ensure that they work the same way even when you upgrade Que in the future:
|
130
129
|
|
131
130
|
```ruby
|
132
|
-
# Update the schema to version #
|
133
|
-
Que.migrate!(version:
|
131
|
+
# Update the schema to version #6.
|
132
|
+
Que.migrate!(version: 6)
|
134
133
|
|
135
134
|
# Remove Que's jobs table entirely.
|
136
135
|
Que.migrate!(version: 0)
|
@@ -442,7 +441,7 @@ que -q default -q credit_cards
|
|
442
441
|
Then you can set jobs to be enqueued in that queue specifically:
|
443
442
|
|
444
443
|
```ruby
|
445
|
-
ProcessCreditCard.enqueue
|
444
|
+
ProcessCreditCard.enqueue(current_user.id, job_options: { queue: 'credit_cards' })
|
446
445
|
|
447
446
|
# Or:
|
448
447
|
|
@@ -456,7 +455,7 @@ end
|
|
456
455
|
In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
|
457
456
|
|
458
457
|
```ruby
|
459
|
-
Que.enqueue
|
458
|
+
Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard', queue: 'credit_cards' })
|
460
459
|
```
|
461
460
|
|
462
461
|
## Shutting Down Safely
|
@@ -550,7 +549,7 @@ require 'que'
|
|
550
549
|
Sequel.migration do
|
551
550
|
up do
|
552
551
|
Que.connection = self
|
553
|
-
Que.migrate!(version:
|
552
|
+
Que.migrate!(version: 6)
|
554
553
|
end
|
555
554
|
down do
|
556
555
|
Que.connection = self
|
@@ -586,7 +585,7 @@ Sequel automatically wraps model persistance actions (create, update, destroy) i
|
|
586
585
|
|
587
586
|
## Using Que With ActiveJob
|
588
587
|
|
589
|
-
You can include `Que::ActiveJob::JobExtensions` into your `ApplicationJob` subclass to get support for all of Que's
|
588
|
+
You can include `Que::ActiveJob::JobExtensions` into your `ApplicationJob` subclass to get support for all of Que's
|
590
589
|
[helper methods](#job-helper-methods). These methods will become no-ops if you use a queue adapter that isn't Que, so if you like to use a different adapter in development they shouldn't interfere.
|
591
590
|
|
592
591
|
Additionally, including `Que::ActiveJob::JobExtensions` lets you define a run() method that supports keyword arguments.
|
@@ -12,8 +12,10 @@ module Que
|
|
12
12
|
end
|
13
13
|
|
14
14
|
def perform(*args)
|
15
|
+
args, kwargs = Que.split_out_ruby2_keywords(args)
|
16
|
+
|
15
17
|
Que.internal_log(:active_job_perform, self) do
|
16
|
-
{args: args}
|
18
|
+
{args: args, kwargs: kwargs}
|
17
19
|
end
|
18
20
|
|
19
21
|
_run(
|
@@ -21,7 +23,12 @@ module Que
|
|
21
23
|
que_filter_args(
|
22
24
|
args.map { |a| a.is_a?(Hash) ? a.deep_symbolize_keys : a }
|
23
25
|
)
|
24
|
-
)
|
26
|
+
),
|
27
|
+
kwargs: Que.recursively_freeze(
|
28
|
+
que_filter_args(
|
29
|
+
kwargs.deep_symbolize_keys,
|
30
|
+
)
|
31
|
+
),
|
25
32
|
)
|
26
33
|
end
|
27
34
|
|
@@ -53,37 +60,46 @@ module Que
|
|
53
60
|
# A module that we mix into ActiveJob's wrapper for Que::Job, to maintain
|
54
61
|
# backwards-compatibility with internal changes we make.
|
55
62
|
module WrapperExtensions
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
60
|
-
|
63
|
+
module ClassMethods
|
64
|
+
# We've dropped support for job options supplied as top-level keywords, but ActiveJob's QueAdapter still uses them. So we have to move them into the job_options hash ourselves.
|
65
|
+
def enqueue(args, priority:, queue:, run_at: nil)
|
66
|
+
super(args, job_options: { priority: priority, queue: queue, run_at: run_at })
|
67
|
+
end
|
61
68
|
end
|
62
69
|
|
63
|
-
|
64
|
-
#
|
65
|
-
#
|
66
|
-
#
|
67
|
-
|
68
|
-
|
70
|
+
module InstanceMethods
|
71
|
+
# The Rails adapter (built against a pre-1.0 version of this gem)
|
72
|
+
# assumes that it can access a job's id via job.attrs["job_id"]. So,
|
73
|
+
# oblige it.
|
74
|
+
def attrs
|
75
|
+
{"job_id" => que_attrs[:id]}
|
76
|
+
end
|
77
|
+
|
78
|
+
def run(args)
|
79
|
+
# Our ActiveJob extensions expect to be able to operate on the actual
|
80
|
+
# job object, but there's no way to access it through ActiveJob. So,
|
81
|
+
# scope it to the current thread. It's a bit messy, but it's the best
|
82
|
+
# option under the circumstances (doesn't require hacking ActiveJob in
|
83
|
+
# any more extensive way).
|
69
84
|
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
85
|
+
# There's no reason this logic should ever nest, because it wouldn't
|
86
|
+
# make sense to run a worker inside of a job, but even so, assert that
|
87
|
+
# nothing absurd is going on.
|
88
|
+
Que.assert NilClass, Thread.current[:que_current_job]
|
74
89
|
|
75
|
-
|
76
|
-
|
90
|
+
begin
|
91
|
+
Thread.current[:que_current_job] = self
|
77
92
|
|
78
|
-
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
|
83
|
-
|
84
|
-
|
85
|
-
|
86
|
-
|
93
|
+
# We symbolize the args hash but ActiveJob doesn't like that :/
|
94
|
+
super(args.deep_stringify_keys)
|
95
|
+
ensure
|
96
|
+
# Also assert that the current job state was only removed now, but
|
97
|
+
# unset the job first so that an assertion failure doesn't mess up
|
98
|
+
# the state any more than it already has.
|
99
|
+
current = Thread.current[:que_current_job]
|
100
|
+
Thread.current[:que_current_job] = nil
|
101
|
+
Que.assert(self, current)
|
102
|
+
end
|
87
103
|
end
|
88
104
|
end
|
89
105
|
end
|
@@ -92,6 +108,7 @@ end
|
|
92
108
|
|
93
109
|
class ActiveJob::QueueAdapters::QueAdapter
|
94
110
|
class JobWrapper < Que::Job
|
95
|
-
|
111
|
+
extend Que::ActiveJob::WrapperExtensions::ClassMethods
|
112
|
+
prepend Que::ActiveJob::WrapperExtensions::InstanceMethods
|
96
113
|
end
|
97
114
|
end
|
@@ -39,8 +39,8 @@ module Que
|
|
39
39
|
where("que_jobs.data @> ?", JSON.dump(tags: [tag]))
|
40
40
|
end
|
41
41
|
|
42
|
-
def by_args(*args)
|
43
|
-
where("que_jobs.args @> ?", JSON.dump(args))
|
42
|
+
def by_args(*args, **kwargs)
|
43
|
+
where("que_jobs.args @> ? AND que_jobs.kwargs @> ?", JSON.dump(args), JSON.dump(kwargs))
|
44
44
|
end
|
45
45
|
end
|
46
46
|
end
|
data/lib/que/job.rb
CHANGED
@@ -12,7 +12,7 @@ module Que
|
|
12
12
|
SQL[:insert_job] =
|
13
13
|
%{
|
14
14
|
INSERT INTO public.que_jobs
|
15
|
-
(queue, priority, run_at, job_class, args, data, job_schema_version)
|
15
|
+
(queue, priority, run_at, job_class, args, kwargs, data, job_schema_version)
|
16
16
|
VALUES
|
17
17
|
(
|
18
18
|
coalesce($1, 'default')::text,
|
@@ -21,6 +21,7 @@ module Que
|
|
21
21
|
$4::text,
|
22
22
|
coalesce($5, '[]')::jsonb,
|
23
23
|
coalesce($6, '{}')::jsonb,
|
24
|
+
coalesce($7, '{}')::jsonb,
|
24
25
|
#{Que.job_schema_version}
|
25
26
|
)
|
26
27
|
RETURNING *
|
@@ -56,13 +57,10 @@ module Que
|
|
56
57
|
:priority,
|
57
58
|
:run_at
|
58
59
|
|
59
|
-
def enqueue(
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
)
|
64
|
-
arg_opts, job_options = _extract_job_options(arg_opts, job_options.dup)
|
65
|
-
args << arg_opts if arg_opts.any?
|
60
|
+
def enqueue(*args)
|
61
|
+
args, kwargs = Que.split_out_ruby2_keywords(args)
|
62
|
+
|
63
|
+
job_options = kwargs.delete(:job_options) || {}
|
66
64
|
|
67
65
|
if job_options[:tags]
|
68
66
|
if job_options[:tags].length > MAXIMUM_TAGS_COUNT
|
@@ -81,6 +79,7 @@ module Que
|
|
81
79
|
priority: job_options[:priority] || resolve_que_setting(:priority),
|
82
80
|
run_at: job_options[:run_at] || resolve_que_setting(:run_at),
|
83
81
|
args: Que.serialize_json(args),
|
82
|
+
kwargs: Que.serialize_json(kwargs),
|
84
83
|
data: job_options[:tags] ? Que.serialize_json(tags: job_options[:tags]) : "{}",
|
85
84
|
job_class: \
|
86
85
|
job_options[:job_class] || name ||
|
@@ -89,27 +88,31 @@ module Que
|
|
89
88
|
|
90
89
|
if attrs[:run_at].nil? && resolve_que_setting(:run_synchronously)
|
91
90
|
attrs[:args] = Que.deserialize_json(attrs[:args])
|
91
|
+
attrs[:kwargs] = Que.deserialize_json(attrs[:kwargs])
|
92
92
|
attrs[:data] = Que.deserialize_json(attrs[:data])
|
93
93
|
_run_attrs(attrs)
|
94
94
|
else
|
95
95
|
values =
|
96
96
|
Que.execute(
|
97
97
|
:insert_job,
|
98
|
-
attrs.values_at(:queue, :priority, :run_at, :job_class, :args, :data),
|
98
|
+
attrs.values_at(:queue, :priority, :run_at, :job_class, :args, :kwargs, :data),
|
99
99
|
).first
|
100
|
-
|
101
100
|
new(values)
|
102
101
|
end
|
103
102
|
end
|
103
|
+
ruby2_keywords(:enqueue) if respond_to?(:ruby2_keywords, true)
|
104
104
|
|
105
105
|
def run(*args)
|
106
106
|
# Make sure things behave the same as they would have with a round-trip
|
107
107
|
# to the DB.
|
108
|
+
args, kwargs = Que.split_out_ruby2_keywords(args)
|
108
109
|
args = Que.deserialize_json(Que.serialize_json(args))
|
110
|
+
kwargs = Que.deserialize_json(Que.serialize_json(kwargs))
|
109
111
|
|
110
112
|
# Should not fail if there's no DB connection.
|
111
|
-
_run_attrs(args: args)
|
113
|
+
_run_attrs(args: args, kwargs: kwargs)
|
112
114
|
end
|
115
|
+
ruby2_keywords(:run) if respond_to?(:ruby2_keywords, true)
|
113
116
|
|
114
117
|
def resolve_que_setting(setting, *args)
|
115
118
|
value = send(setting) if respond_to?(setting)
|
@@ -136,27 +139,6 @@ module Que
|
|
136
139
|
end
|
137
140
|
end
|
138
141
|
end
|
139
|
-
|
140
|
-
def _extract_job_options(arg_opts, job_options)
|
141
|
-
deprecated_job_option_names = []
|
142
|
-
|
143
|
-
%i[queue priority run_at job_class tags].each do |option_name|
|
144
|
-
next unless arg_opts.key?(option_name) && job_options[option_name].nil?
|
145
|
-
|
146
|
-
job_options[option_name] = arg_opts.delete(option_name)
|
147
|
-
deprecated_job_option_names << option_name
|
148
|
-
end
|
149
|
-
|
150
|
-
_log_job_options_deprecation(deprecated_job_option_names)
|
151
|
-
|
152
|
-
[arg_opts, job_options]
|
153
|
-
end
|
154
|
-
|
155
|
-
def _log_job_options_deprecation(deprecated_job_option_names)
|
156
|
-
return unless deprecated_job_option_names.any?
|
157
|
-
|
158
|
-
warn "Passing job options like (#{deprecated_job_option_names.join(', ')}) to `JobClass.enqueue` as top level keyword args has been deprecated and will be removed in version 2.0. Please wrap job options in an explicit `job_options` keyword arg instead."
|
159
|
-
end
|
160
142
|
end
|
161
143
|
|
162
144
|
# Set up some defaults.
|
data/lib/que/job_buffer.rb
CHANGED
@@ -6,7 +6,7 @@
|
|
6
6
|
|
7
7
|
module Que
|
8
8
|
class JobBuffer
|
9
|
-
attr_reader :maximum_size, :
|
9
|
+
attr_reader :maximum_size, :priority_queues
|
10
10
|
|
11
11
|
# Since we use a mutex, which is not reentrant, we have to be a little
|
12
12
|
# careful to not call a method that locks the mutex when we've already
|
@@ -17,20 +17,11 @@ module Que
|
|
17
17
|
|
18
18
|
def initialize(
|
19
19
|
maximum_size:,
|
20
|
-
minimum_size:,
|
21
20
|
priorities:
|
22
21
|
)
|
23
22
|
@maximum_size = Que.assert(Integer, maximum_size)
|
24
23
|
Que.assert(maximum_size >= 0) { "maximum_size for a JobBuffer must be at least zero!" }
|
25
24
|
|
26
|
-
@minimum_size = Que.assert(Integer, minimum_size)
|
27
|
-
Que.assert(minimum_size >= 0) { "minimum_size for a JobBuffer must be at least zero!" }
|
28
|
-
|
29
|
-
Que.assert(minimum_size <= maximum_size) do
|
30
|
-
"minimum buffer size (#{minimum_size}) is " \
|
31
|
-
"greater than the maximum buffer size (#{maximum_size})!"
|
32
|
-
end
|
33
|
-
|
34
25
|
@stop = false
|
35
26
|
@array = []
|
36
27
|
@mutex = Mutex.new
|
data/lib/que/job_methods.rb
CHANGED
@@ -39,12 +39,16 @@ module Que
|
|
39
39
|
# Run the job with error handling and cleanup logic. Optionally support
|
40
40
|
# overriding the args, because it's necessary when jobs are invoked from
|
41
41
|
# ActiveJob.
|
42
|
-
def _run(args: nil, reraise_errors: false)
|
42
|
+
def _run(args: nil, kwargs: nil, reraise_errors: false)
|
43
43
|
if args.nil? && que_target
|
44
44
|
args = que_target.que_attrs.fetch(:args)
|
45
45
|
end
|
46
46
|
|
47
|
-
|
47
|
+
if kwargs.nil? && que_target
|
48
|
+
kwargs = que_target.que_attrs.fetch(:kwargs)
|
49
|
+
end
|
50
|
+
|
51
|
+
run(*args, **kwargs)
|
48
52
|
default_resolve_action if que_target && !que_target.que_resolved
|
49
53
|
rescue => error
|
50
54
|
raise error unless que_target
|
data/lib/que/locker.rb
CHANGED
@@ -45,7 +45,6 @@ module Que
|
|
45
45
|
|
46
46
|
DEFAULT_POLL_INTERVAL = 5.0
|
47
47
|
DEFAULT_WAIT_PERIOD = 50
|
48
|
-
DEFAULT_MINIMUM_BUFFER_SIZE = 2
|
49
48
|
DEFAULT_MAXIMUM_BUFFER_SIZE = 8
|
50
49
|
DEFAULT_WORKER_PRIORITIES = [10, 30, 50, nil, nil, nil].freeze
|
51
50
|
|
@@ -57,7 +56,6 @@ module Que
|
|
57
56
|
poll_interval: DEFAULT_POLL_INTERVAL,
|
58
57
|
wait_period: DEFAULT_WAIT_PERIOD,
|
59
58
|
maximum_buffer_size: DEFAULT_MAXIMUM_BUFFER_SIZE,
|
60
|
-
minimum_buffer_size: DEFAULT_MINIMUM_BUFFER_SIZE,
|
61
59
|
worker_priorities: DEFAULT_WORKER_PRIORITIES,
|
62
60
|
on_worker_start: nil
|
63
61
|
)
|
@@ -77,7 +75,6 @@ module Que
|
|
77
75
|
# ResultQueue to receive messages from workers.
|
78
76
|
@job_buffer = JobBuffer.new(
|
79
77
|
maximum_size: maximum_buffer_size,
|
80
|
-
minimum_size: minimum_buffer_size,
|
81
78
|
priorities: worker_priorities.uniq,
|
82
79
|
)
|
83
80
|
|
@@ -93,7 +90,6 @@ module Que
|
|
93
90
|
poll_interval: poll_interval,
|
94
91
|
wait_period: wait_period,
|
95
92
|
maximum_buffer_size: maximum_buffer_size,
|
96
|
-
minimum_buffer_size: minimum_buffer_size,
|
97
93
|
worker_priorities: worker_priorities,
|
98
94
|
}
|
99
95
|
end
|
@@ -0,0 +1,8 @@
|
|
1
|
+
DROP INDEX que_jobs_kwargs_gin_idx;
|
2
|
+
ALTER TABLE que_jobs DROP COLUMN kwargs;
|
3
|
+
|
4
|
+
ALTER INDEX que_poll_idx RENAME TO que_poll_idx_with_job_schema_version;
|
5
|
+
CREATE INDEX que_poll_idx ON que_jobs (queue, priority, run_at, id) WHERE (finished_at IS NULL AND expired_at IS NULL);
|
6
|
+
|
7
|
+
ALTER TABLE que_jobs ALTER COLUMN job_schema_version SET DEFAULT 1;
|
8
|
+
ALTER TABLE que_jobs ALTER COLUMN job_schema_version DROP NOT NULL;
|
@@ -0,0 +1,8 @@
|
|
1
|
+
ALTER TABLE que_jobs ADD COLUMN kwargs JSONB NOT NULL DEFAULT '{}';
|
2
|
+
CREATE INDEX que_jobs_kwargs_gin_idx ON que_jobs USING gin (kwargs jsonb_path_ops);
|
3
|
+
|
4
|
+
DROP INDEX que_poll_idx;
|
5
|
+
ALTER INDEX que_poll_idx_with_job_schema_version RENAME TO que_poll_idx;
|
6
|
+
|
7
|
+
ALTER TABLE que_jobs ALTER COLUMN job_schema_version DROP DEFAULT;
|
8
|
+
ALTER TABLE que_jobs ALTER COLUMN job_schema_version SET NOT NULL;
|
data/lib/que/migrations.rb
CHANGED
data/lib/que/poller.rb
CHANGED
@@ -146,8 +146,6 @@ module Que
|
|
146
146
|
|
147
147
|
return unless should_poll?
|
148
148
|
|
149
|
-
expected_count = priorities.inject(0){|s,(_,c)| s + c}
|
150
|
-
|
151
149
|
jobs =
|
152
150
|
connection.execute_prepared(
|
153
151
|
:poll_jobs,
|
@@ -159,7 +157,7 @@ module Que
|
|
159
157
|
)
|
160
158
|
|
161
159
|
@last_polled_at = Time.now
|
162
|
-
@last_poll_satisfied =
|
160
|
+
@last_poll_satisfied = poll_satisfied?(priorities, jobs)
|
163
161
|
|
164
162
|
Que.internal_log :poller_polled, self do
|
165
163
|
{
|
@@ -265,5 +263,12 @@ module Que
|
|
265
263
|
SQL
|
266
264
|
end
|
267
265
|
end
|
266
|
+
|
267
|
+
private
|
268
|
+
|
269
|
+
def poll_satisfied?(priorities, jobs)
|
270
|
+
lowest_priority = priorities.keys.max
|
271
|
+
jobs.count >= priorities[lowest_priority]
|
272
|
+
end
|
268
273
|
end
|
269
274
|
end
|
data/lib/que/sequel/model.rb
CHANGED
@@ -40,8 +40,11 @@ module Que
|
|
40
40
|
where(QUALIFIED_TABLE[:data].pg_jsonb.contains(JSON.dump(tags: [tag])))
|
41
41
|
end
|
42
42
|
|
43
|
-
def by_args(*args)
|
44
|
-
where(
|
43
|
+
def by_args(*args, **kwargs)
|
44
|
+
where(
|
45
|
+
QUALIFIED_TABLE[:args].pg_jsonb.contains(JSON.dump(args)) &
|
46
|
+
QUALIFIED_TABLE[:kwargs].pg_jsonb.contains(JSON.dump(kwargs))
|
47
|
+
)
|
45
48
|
end
|
46
49
|
end
|
47
50
|
end
|
@@ -0,0 +1,19 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
# Temporary module allowing ruby2 keyword args to be extracted from an *args splat
|
4
|
+
# Allows us to ensure consistent behaviour when running on ruby 2 vs ruby 3
|
5
|
+
# We can remove this if/when we drop support for ruby 2
|
6
|
+
|
7
|
+
require 'json'
|
8
|
+
|
9
|
+
module Que
|
10
|
+
module Utils
|
11
|
+
module Ruby2Keywords
|
12
|
+
def split_out_ruby2_keywords(args)
|
13
|
+
return [args, {}] unless args.last&.is_a?(Hash) && Hash.ruby2_keywords_hash?(args.last)
|
14
|
+
|
15
|
+
[args[0..-2], args.last]
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
data/lib/que/version.rb
CHANGED
data/lib/que/worker.rb
CHANGED
data/lib/que.rb
CHANGED
@@ -29,8 +29,11 @@ module Que
|
|
29
29
|
require_relative 'que/utils/logging'
|
30
30
|
require_relative 'que/utils/middleware'
|
31
31
|
require_relative 'que/utils/queue_management'
|
32
|
+
require_relative 'que/utils/ruby2_keywords'
|
32
33
|
require_relative 'que/utils/transactions'
|
33
34
|
|
35
|
+
require_relative 'que/version'
|
36
|
+
|
34
37
|
require_relative 'que/connection'
|
35
38
|
require_relative 'que/connection_pool'
|
36
39
|
require_relative 'que/job_methods'
|
@@ -41,7 +44,6 @@ module Que
|
|
41
44
|
require_relative 'que/migrations'
|
42
45
|
require_relative 'que/poller'
|
43
46
|
require_relative 'que/result_queue'
|
44
|
-
require_relative 'que/version'
|
45
47
|
require_relative 'que/worker'
|
46
48
|
|
47
49
|
class << self
|
@@ -60,6 +62,7 @@ module Que
|
|
60
62
|
include Utils::Logging
|
61
63
|
include Utils::Middleware
|
62
64
|
include Utils::QueueManagement
|
65
|
+
include Utils::Ruby2Keywords
|
63
66
|
include Utils::Transactions
|
64
67
|
|
65
68
|
extend Forwardable
|