que 1.3.0 → 2.0.0.beta1

Sign up to get free protection for your applications and to get access to all the features.
data/Dockerfile CHANGED
@@ -1,4 +1,4 @@
1
- FROM ruby:2.7.5-slim-buster@sha256:4cbbe2fba099026b243200aa8663f56476950cc64ccd91d7aaccddca31e445b5 AS base
1
+ FROM ruby:3.1.1-slim-buster@sha256:2ada3e4fe7b1703c9333ad4eb9fc12c1d4d60bce0f981281b2151057e928d9ad AS base
2
2
 
3
3
  # Install libpq-dev in our base layer, as it's needed in all environments
4
4
  RUN apt-get update \
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Que ![tests](https://github.com/que-rb/que/workflows/tests/badge.svg)
2
2
 
3
- **This README and the rest of the docs on the master branch all refer to Que 1.0. If you're using version 0.x, please refer to the docs on [the 0.x branch](https://github.com/que-rb/que/tree/0.x).**
3
+ **This README and the rest of the docs on the master branch all refer to Que 2.x. For older versions, please refer to the docs on the respective branches: [1.x](https://github.com/que-rb/que/tree/1.x), or [0.x](https://github.com/que-rb/que/tree/0.x).**
4
4
 
5
5
  *TL;DR: Que is a high-performance job queue that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.*
6
6
 
@@ -23,9 +23,9 @@ Que's secondary goal is performance. The worker process is multithreaded, so tha
23
23
 
24
24
  Compatibility:
25
25
 
26
- - MRI Ruby 2.2+
26
+ - MRI Ruby 2.7+
27
27
  - PostgreSQL 9.5+
28
- - Rails 4.1+ (optional)
28
+ - Rails 6.0+ (optional)
29
29
 
30
30
  **Please note** - Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.
31
31
 
@@ -54,12 +54,12 @@ gem install que
54
54
  First, create the queue schema in a migration. For example:
55
55
 
56
56
  ```ruby
57
- class CreateQueSchema < ActiveRecord::Migration[5.0]
57
+ class CreateQueSchema < ActiveRecord::Migration[6.0]
58
58
  def up
59
59
  # Whenever you use Que in a migration, always specify the version you're
60
60
  # migrating to. If you're unsure what the current version is, check the
61
61
  # changelog.
62
- Que.migrate!(version: 5)
62
+ Que.migrate!(version: 6)
63
63
  end
64
64
 
65
65
  def down
@@ -117,10 +117,10 @@ end
117
117
  You can also add options to run the job after a specific time, or with a specific priority:
118
118
 
119
119
  ```ruby
120
- ChargeCreditCard.enqueue card.id, user_id: current_user.id, run_at: 1.day.from_now, priority: 5
120
+ ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 })
121
121
  ```
122
122
  ## Running the Que Worker
123
- In order to process jobs, you must start a separate worker process outside of your main server.
123
+ In order to process jobs, you must start a separate worker process outside of your main server.
124
124
 
125
125
  ```bash
126
126
  bundle exec que
@@ -142,7 +142,7 @@ You may need to pass que a file path to require so that it can load your app. Qu
142
142
 
143
143
  If you're using ActiveRecord to dump your database's schema, please [set your schema_format to :sql](http://guides.rubyonrails.org/migrations.html#types-of-schema-dumps) so that Que's table structure is managed correctly. This is a good idea regardless, as the `:ruby` schema format doesn't support many of PostgreSQL's advanced features.
144
144
 
145
- Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. In 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default. You can either:
145
+ Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. As of 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default. You can either:
146
146
 
147
147
  - [Configure Rails](https://guides.rubyonrails.org/configuring.html) to send all internal job types to the 'default' queue by adding the following to `config/application.rb`:
148
148
 
@@ -138,10 +138,9 @@ module Que
138
138
  opts.on(
139
139
  '--minimum-buffer-size [SIZE]',
140
140
  Integer,
141
- "Set minimum number of jobs to be locked and held in this " \
142
- "process awaiting a worker (default: 2)",
141
+ "Unused (deprecated)",
143
142
  ) do |s|
144
- options[:minimum_buffer_size] = s
143
+ warn "The --minimum-buffer-size SIZE option has been deprecated and will be removed in v2.0 (it's actually been unused since v1.0.0.beta4)"
145
144
  end
146
145
 
147
146
  opts.on(
@@ -221,7 +220,7 @@ OUTPUT
221
220
 
222
221
  locker =
223
222
  begin
224
- Que::Locker.new(options)
223
+ Que::Locker.new(**options)
225
224
  rescue => e
226
225
  output.puts(e.message)
227
226
  return 1
@@ -236,7 +235,7 @@ OUTPUT
236
235
  <<~STARTUP
237
236
  Que #{Que::VERSION} started worker process with:
238
237
  Worker threads: #{locker.workers.length} (priorities: #{locker.workers.map { |w| w.priority || 'any' }.join(', ')})
239
- Buffer size: #{locker.job_buffer.minimum_size}-#{locker.job_buffer.maximum_size}
238
+ Buffer size: #{locker.job_buffer.maximum_size}
240
239
  Queues:
241
240
  #{locker.queues.map { |queue, interval| " - #{queue} (poll interval: #{interval}s)" }.join("\n")}
242
241
  Que waiting for jobs...
data/docker-compose.yml CHANGED
@@ -10,13 +10,14 @@ services:
10
10
  - db
11
11
  volumes:
12
12
  - .:/work
13
- - ruby-2.7.5-gem-cache:/usr/local/bundle
13
+ - ruby-3.1.1-gem-cache:/usr/local/bundle
14
14
  - ~/.docker-rc.d/:/.docker-rc.d/:ro
15
15
  working_dir: /work
16
16
  entrypoint: /work/scripts/docker-entrypoint
17
17
  command: bash
18
18
  environment:
19
19
  DATABASE_URL: postgres://que:que@db/que-test
20
+ USE_RAILS: ~
20
21
 
21
22
  db:
22
23
  image: "postgres:${POSTGRES_VERSION-13}"
@@ -43,4 +44,4 @@ services:
43
44
 
44
45
  volumes:
45
46
  db-data: ~
46
- ruby-2.7.5-gem-cache: ~
47
+ ruby-3.1.1-gem-cache: ~
data/docs/README.md CHANGED
@@ -4,7 +4,7 @@ Docs Index
4
4
  - [Command Line Interface](#command-line-interface)
5
5
  * [worker-priorities and worker-count](#worker-priorities-and-worker-count)
6
6
  * [poll-interval](#poll-interval)
7
- * [minimum-buffer-size and maximum-buffer-size](#minimum-buffer-size-and-maximum-buffer-size)
7
+ * [maximum-buffer-size](#maximum-buffer-size)
8
8
  * [connection-url](#connection-url)
9
9
  * [wait-period](#wait-period)
10
10
  * [log-internals](#log-internals)
@@ -62,7 +62,6 @@ usage: que [options] [file/to/require] ...
62
62
  --connection-url [URL] Set a custom database url to connect to for locking purposes.
63
63
  --log-internals Log verbosely about Que's internal state. Only recommended for debugging issues
64
64
  --maximum-buffer-size [SIZE] Set maximum number of jobs to be locked and held in this process awaiting a worker (default: 8)
65
- --minimum-buffer-size [SIZE] Set minimum number of jobs to be locked and held in this process awaiting a worker (default: 2)
66
65
  --wait-period [PERIOD] Set maximum interval between checks of the in-memory job queue, in milliseconds (default: 50)
67
66
  ```
68
67
 
@@ -82,9 +81,9 @@ If you pass both worker-count and worker-priorities, the count will trim or pad
82
81
 
83
82
  This option sets the number of seconds the process will wait between polls of the job queue. Jobs that are ready to be worked immediately will be broadcast via the LISTEN/NOTIFY system, so polling is unnecessary for them - polling is only necessary for jobs that are scheduled in the future or which are being delayed due to errors. The default is 5 seconds.
84
83
 
85
- ### minimum-buffer-size and maximum-buffer-size
84
+ ### maximum-buffer-size
86
85
 
87
- These options set the size of the internal buffer that Que uses to hold jobs until they're ready for workers. The default minimum is 2 and the maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked, and will only resort to polling if the buffer dips below 2. If you don't want jobs to be buffered at all, you can set both of these values to zero.
86
+ This option sets the size of the internal buffer that Que uses to hold jobs until they're ready for workers. The default maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked. If you don't want jobs to be buffered at all, you can set this value to zero.
88
87
 
89
88
  ### connection-url
90
89
 
@@ -129,8 +128,8 @@ There are other docs to read if you're using [Sequel](#using-sequel) or [plain P
129
128
  After you've connected Que to the database, you can manage the jobs table. You'll want to migrate to a specific version in a migration file, to ensure that they work the same way even when you upgrade Que in the future:
130
129
 
131
130
  ```ruby
132
- # Update the schema to version #5.
133
- Que.migrate!(version: 5)
131
+ # Update the schema to version #6.
132
+ Que.migrate!(version: 6)
134
133
 
135
134
  # Remove Que's jobs table entirely.
136
135
  Que.migrate!(version: 0)
@@ -442,7 +441,7 @@ que -q default -q credit_cards
442
441
  Then you can set jobs to be enqueued in that queue specifically:
443
442
 
444
443
  ```ruby
445
- ProcessCreditCard.enqueue current_user.id, queue: 'credit_cards'
444
+ ProcessCreditCard.enqueue(current_user.id, job_options: { queue: 'credit_cards' })
446
445
 
447
446
  # Or:
448
447
 
@@ -456,7 +455,7 @@ end
456
455
  In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
457
456
 
458
457
  ```ruby
459
- Que.enqueue current_user.id, job_class: 'ProcessCreditCard', queue: 'credit_cards'
458
+ Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard', queue: 'credit_cards' })
460
459
  ```
461
460
 
462
461
  ## Shutting Down Safely
@@ -550,7 +549,7 @@ require 'que'
550
549
  Sequel.migration do
551
550
  up do
552
551
  Que.connection = self
553
- Que.migrate!(version: 5)
552
+ Que.migrate!(version: 6)
554
553
  end
555
554
  down do
556
555
  Que.connection = self
@@ -586,7 +585,7 @@ Sequel automatically wraps model persistance actions (create, update, destroy) i
586
585
 
587
586
  ## Using Que With ActiveJob
588
587
 
589
- You can include `Que::ActiveJob::JobExtensions` into your `ApplicationJob` subclass to get support for all of Que's
588
+ You can include `Que::ActiveJob::JobExtensions` into your `ApplicationJob` subclass to get support for all of Que's
590
589
  [helper methods](#job-helper-methods). These methods will become no-ops if you use a queue adapter that isn't Que, so if you like to use a different adapter in development they shouldn't interfere.
591
590
 
592
591
  Additionally, including `Que::ActiveJob::JobExtensions` lets you define a run() method that supports keyword arguments.
@@ -12,8 +12,10 @@ module Que
12
12
  end
13
13
 
14
14
  def perform(*args)
15
+ args, kwargs = Que.split_out_ruby2_keywords(args)
16
+
15
17
  Que.internal_log(:active_job_perform, self) do
16
- {args: args}
18
+ {args: args, kwargs: kwargs}
17
19
  end
18
20
 
19
21
  _run(
@@ -21,7 +23,12 @@ module Que
21
23
  que_filter_args(
22
24
  args.map { |a| a.is_a?(Hash) ? a.deep_symbolize_keys : a }
23
25
  )
24
- )
26
+ ),
27
+ kwargs: Que.recursively_freeze(
28
+ que_filter_args(
29
+ kwargs.deep_symbolize_keys,
30
+ )
31
+ ),
25
32
  )
26
33
  end
27
34
 
@@ -53,37 +60,46 @@ module Que
53
60
  # A module that we mix into ActiveJob's wrapper for Que::Job, to maintain
54
61
  # backwards-compatibility with internal changes we make.
55
62
  module WrapperExtensions
56
- # The Rails adapter (built against a pre-1.0 version of this gem)
57
- # assumes that it can access a job's id via job.attrs["job_id"]. So,
58
- # oblige it.
59
- def attrs
60
- {"job_id" => que_attrs[:id]}
63
+ module ClassMethods
64
+ # We've dropped support for job options supplied as top-level keywords, but ActiveJob's QueAdapter still uses them. So we have to move them into the job_options hash ourselves.
65
+ def enqueue(args, priority:, queue:, run_at: nil)
66
+ super(args, job_options: { priority: priority, queue: queue, run_at: run_at })
67
+ end
61
68
  end
62
69
 
63
- def run(args)
64
- # Our ActiveJob extensions expect to be able to operate on the actual
65
- # job object, but there's no way to access it through ActiveJob. So,
66
- # scope it to the current thread. It's a bit messy, but it's the best
67
- # option under the circumstances (doesn't require hacking ActiveJob in
68
- # any more extensive way).
70
+ module InstanceMethods
71
+ # The Rails adapter (built against a pre-1.0 version of this gem)
72
+ # assumes that it can access a job's id via job.attrs["job_id"]. So,
73
+ # oblige it.
74
+ def attrs
75
+ {"job_id" => que_attrs[:id]}
76
+ end
77
+
78
+ def run(args)
79
+ # Our ActiveJob extensions expect to be able to operate on the actual
80
+ # job object, but there's no way to access it through ActiveJob. So,
81
+ # scope it to the current thread. It's a bit messy, but it's the best
82
+ # option under the circumstances (doesn't require hacking ActiveJob in
83
+ # any more extensive way).
69
84
 
70
- # There's no reason this logic should ever nest, because it wouldn't
71
- # make sense to run a worker inside of a job, but even so, assert that
72
- # nothing absurd is going on.
73
- Que.assert NilClass, Thread.current[:que_current_job]
85
+ # There's no reason this logic should ever nest, because it wouldn't
86
+ # make sense to run a worker inside of a job, but even so, assert that
87
+ # nothing absurd is going on.
88
+ Que.assert NilClass, Thread.current[:que_current_job]
74
89
 
75
- begin
76
- Thread.current[:que_current_job] = self
90
+ begin
91
+ Thread.current[:que_current_job] = self
77
92
 
78
- # We symbolize the args hash but ActiveJob doesn't like that :/
79
- super(args.deep_stringify_keys)
80
- ensure
81
- # Also assert that the current job state was only removed now, but
82
- # unset the job first so that an assertion failure doesn't mess up
83
- # the state any more than it already has.
84
- current = Thread.current[:que_current_job]
85
- Thread.current[:que_current_job] = nil
86
- Que.assert(self, current)
93
+ # We symbolize the args hash but ActiveJob doesn't like that :/
94
+ super(args.deep_stringify_keys)
95
+ ensure
96
+ # Also assert that the current job state was only removed now, but
97
+ # unset the job first so that an assertion failure doesn't mess up
98
+ # the state any more than it already has.
99
+ current = Thread.current[:que_current_job]
100
+ Thread.current[:que_current_job] = nil
101
+ Que.assert(self, current)
102
+ end
87
103
  end
88
104
  end
89
105
  end
@@ -92,6 +108,7 @@ end
92
108
 
93
109
  class ActiveJob::QueueAdapters::QueAdapter
94
110
  class JobWrapper < Que::Job
95
- prepend Que::ActiveJob::WrapperExtensions
111
+ extend Que::ActiveJob::WrapperExtensions::ClassMethods
112
+ prepend Que::ActiveJob::WrapperExtensions::InstanceMethods
96
113
  end
97
114
  end
@@ -39,8 +39,8 @@ module Que
39
39
  where("que_jobs.data @> ?", JSON.dump(tags: [tag]))
40
40
  end
41
41
 
42
- def by_args(*args)
43
- where("que_jobs.args @> ?", JSON.dump(args))
42
+ def by_args(*args, **kwargs)
43
+ where("que_jobs.args @> ? AND que_jobs.kwargs @> ?", JSON.dump(args), JSON.dump(kwargs))
44
44
  end
45
45
  end
46
46
  end
data/lib/que/job.rb CHANGED
@@ -12,7 +12,7 @@ module Que
12
12
  SQL[:insert_job] =
13
13
  %{
14
14
  INSERT INTO public.que_jobs
15
- (queue, priority, run_at, job_class, args, data, job_schema_version)
15
+ (queue, priority, run_at, job_class, args, kwargs, data, job_schema_version)
16
16
  VALUES
17
17
  (
18
18
  coalesce($1, 'default')::text,
@@ -21,6 +21,7 @@ module Que
21
21
  $4::text,
22
22
  coalesce($5, '[]')::jsonb,
23
23
  coalesce($6, '{}')::jsonb,
24
+ coalesce($7, '{}')::jsonb,
24
25
  #{Que.job_schema_version}
25
26
  )
26
27
  RETURNING *
@@ -56,13 +57,10 @@ module Que
56
57
  :priority,
57
58
  :run_at
58
59
 
59
- def enqueue(
60
- *args,
61
- job_options: {},
62
- **arg_opts
63
- )
64
- arg_opts, job_options = _extract_job_options(arg_opts, job_options.dup)
65
- args << arg_opts if arg_opts.any?
60
+ def enqueue(*args)
61
+ args, kwargs = Que.split_out_ruby2_keywords(args)
62
+
63
+ job_options = kwargs.delete(:job_options) || {}
66
64
 
67
65
  if job_options[:tags]
68
66
  if job_options[:tags].length > MAXIMUM_TAGS_COUNT
@@ -81,6 +79,7 @@ module Que
81
79
  priority: job_options[:priority] || resolve_que_setting(:priority),
82
80
  run_at: job_options[:run_at] || resolve_que_setting(:run_at),
83
81
  args: Que.serialize_json(args),
82
+ kwargs: Que.serialize_json(kwargs),
84
83
  data: job_options[:tags] ? Que.serialize_json(tags: job_options[:tags]) : "{}",
85
84
  job_class: \
86
85
  job_options[:job_class] || name ||
@@ -89,27 +88,31 @@ module Que
89
88
 
90
89
  if attrs[:run_at].nil? && resolve_que_setting(:run_synchronously)
91
90
  attrs[:args] = Que.deserialize_json(attrs[:args])
91
+ attrs[:kwargs] = Que.deserialize_json(attrs[:kwargs])
92
92
  attrs[:data] = Que.deserialize_json(attrs[:data])
93
93
  _run_attrs(attrs)
94
94
  else
95
95
  values =
96
96
  Que.execute(
97
97
  :insert_job,
98
- attrs.values_at(:queue, :priority, :run_at, :job_class, :args, :data),
98
+ attrs.values_at(:queue, :priority, :run_at, :job_class, :args, :kwargs, :data),
99
99
  ).first
100
-
101
100
  new(values)
102
101
  end
103
102
  end
103
+ ruby2_keywords(:enqueue) if respond_to?(:ruby2_keywords, true)
104
104
 
105
105
  def run(*args)
106
106
  # Make sure things behave the same as they would have with a round-trip
107
107
  # to the DB.
108
+ args, kwargs = Que.split_out_ruby2_keywords(args)
108
109
  args = Que.deserialize_json(Que.serialize_json(args))
110
+ kwargs = Que.deserialize_json(Que.serialize_json(kwargs))
109
111
 
110
112
  # Should not fail if there's no DB connection.
111
- _run_attrs(args: args)
113
+ _run_attrs(args: args, kwargs: kwargs)
112
114
  end
115
+ ruby2_keywords(:run) if respond_to?(:ruby2_keywords, true)
113
116
 
114
117
  def resolve_que_setting(setting, *args)
115
118
  value = send(setting) if respond_to?(setting)
@@ -136,27 +139,6 @@ module Que
136
139
  end
137
140
  end
138
141
  end
139
-
140
- def _extract_job_options(arg_opts, job_options)
141
- deprecated_job_option_names = []
142
-
143
- %i[queue priority run_at job_class tags].each do |option_name|
144
- next unless arg_opts.key?(option_name) && job_options[option_name].nil?
145
-
146
- job_options[option_name] = arg_opts.delete(option_name)
147
- deprecated_job_option_names << option_name
148
- end
149
-
150
- _log_job_options_deprecation(deprecated_job_option_names)
151
-
152
- [arg_opts, job_options]
153
- end
154
-
155
- def _log_job_options_deprecation(deprecated_job_option_names)
156
- return unless deprecated_job_option_names.any?
157
-
158
- warn "Passing job options like (#{deprecated_job_option_names.join(', ')}) to `JobClass.enqueue` as top level keyword args has been deprecated and will be removed in version 2.0. Please wrap job options in an explicit `job_options` keyword arg instead."
159
- end
160
142
  end
161
143
 
162
144
  # Set up some defaults.
@@ -6,7 +6,7 @@
6
6
 
7
7
  module Que
8
8
  class JobBuffer
9
- attr_reader :maximum_size, :minimum_size, :priority_queues
9
+ attr_reader :maximum_size, :priority_queues
10
10
 
11
11
  # Since we use a mutex, which is not reentrant, we have to be a little
12
12
  # careful to not call a method that locks the mutex when we've already
@@ -17,20 +17,11 @@ module Que
17
17
 
18
18
  def initialize(
19
19
  maximum_size:,
20
- minimum_size:,
21
20
  priorities:
22
21
  )
23
22
  @maximum_size = Que.assert(Integer, maximum_size)
24
23
  Que.assert(maximum_size >= 0) { "maximum_size for a JobBuffer must be at least zero!" }
25
24
 
26
- @minimum_size = Que.assert(Integer, minimum_size)
27
- Que.assert(minimum_size >= 0) { "minimum_size for a JobBuffer must be at least zero!" }
28
-
29
- Que.assert(minimum_size <= maximum_size) do
30
- "minimum buffer size (#{minimum_size}) is " \
31
- "greater than the maximum buffer size (#{maximum_size})!"
32
- end
33
-
34
25
  @stop = false
35
26
  @array = []
36
27
  @mutex = Mutex.new
@@ -39,12 +39,16 @@ module Que
39
39
  # Run the job with error handling and cleanup logic. Optionally support
40
40
  # overriding the args, because it's necessary when jobs are invoked from
41
41
  # ActiveJob.
42
- def _run(args: nil, reraise_errors: false)
42
+ def _run(args: nil, kwargs: nil, reraise_errors: false)
43
43
  if args.nil? && que_target
44
44
  args = que_target.que_attrs.fetch(:args)
45
45
  end
46
46
 
47
- run(*args)
47
+ if kwargs.nil? && que_target
48
+ kwargs = que_target.que_attrs.fetch(:kwargs)
49
+ end
50
+
51
+ run(*args, **kwargs)
48
52
  default_resolve_action if que_target && !que_target.que_resolved
49
53
  rescue => error
50
54
  raise error unless que_target
data/lib/que/locker.rb CHANGED
@@ -45,7 +45,6 @@ module Que
45
45
 
46
46
  DEFAULT_POLL_INTERVAL = 5.0
47
47
  DEFAULT_WAIT_PERIOD = 50
48
- DEFAULT_MINIMUM_BUFFER_SIZE = 2
49
48
  DEFAULT_MAXIMUM_BUFFER_SIZE = 8
50
49
  DEFAULT_WORKER_PRIORITIES = [10, 30, 50, nil, nil, nil].freeze
51
50
 
@@ -57,7 +56,6 @@ module Que
57
56
  poll_interval: DEFAULT_POLL_INTERVAL,
58
57
  wait_period: DEFAULT_WAIT_PERIOD,
59
58
  maximum_buffer_size: DEFAULT_MAXIMUM_BUFFER_SIZE,
60
- minimum_buffer_size: DEFAULT_MINIMUM_BUFFER_SIZE,
61
59
  worker_priorities: DEFAULT_WORKER_PRIORITIES,
62
60
  on_worker_start: nil
63
61
  )
@@ -77,7 +75,6 @@ module Que
77
75
  # ResultQueue to receive messages from workers.
78
76
  @job_buffer = JobBuffer.new(
79
77
  maximum_size: maximum_buffer_size,
80
- minimum_size: minimum_buffer_size,
81
78
  priorities: worker_priorities.uniq,
82
79
  )
83
80
 
@@ -93,7 +90,6 @@ module Que
93
90
  poll_interval: poll_interval,
94
91
  wait_period: wait_period,
95
92
  maximum_buffer_size: maximum_buffer_size,
96
- minimum_buffer_size: minimum_buffer_size,
97
93
  worker_priorities: worker_priorities,
98
94
  }
99
95
  end
@@ -0,0 +1,8 @@
1
+ DROP INDEX que_jobs_kwargs_gin_idx;
2
+ ALTER TABLE que_jobs DROP COLUMN kwargs;
3
+
4
+ ALTER INDEX que_poll_idx RENAME TO que_poll_idx_with_job_schema_version;
5
+ CREATE INDEX que_poll_idx ON que_jobs (queue, priority, run_at, id) WHERE (finished_at IS NULL AND expired_at IS NULL);
6
+
7
+ ALTER TABLE que_jobs ALTER COLUMN job_schema_version SET DEFAULT 1;
8
+ ALTER TABLE que_jobs ALTER COLUMN job_schema_version DROP NOT NULL;
@@ -0,0 +1,8 @@
1
+ ALTER TABLE que_jobs ADD COLUMN kwargs JSONB NOT NULL DEFAULT '{}';
2
+ CREATE INDEX que_jobs_kwargs_gin_idx ON que_jobs USING gin (kwargs jsonb_path_ops);
3
+
4
+ DROP INDEX que_poll_idx;
5
+ ALTER INDEX que_poll_idx_with_job_schema_version RENAME TO que_poll_idx;
6
+
7
+ ALTER TABLE que_jobs ALTER COLUMN job_schema_version DROP DEFAULT;
8
+ ALTER TABLE que_jobs ALTER COLUMN job_schema_version SET NOT NULL;
@@ -4,7 +4,7 @@ module Que
4
4
  module Migrations
5
5
  # In order to ship a schema change, add the relevant up and down sql files
6
6
  # to the migrations directory, and bump the version here.
7
- CURRENT_VERSION = 5
7
+ CURRENT_VERSION = 6
8
8
 
9
9
  class << self
10
10
  def migrate!(version:)
data/lib/que/poller.rb CHANGED
@@ -146,8 +146,6 @@ module Que
146
146
 
147
147
  return unless should_poll?
148
148
 
149
- expected_count = priorities.inject(0){|s,(_,c)| s + c}
150
-
151
149
  jobs =
152
150
  connection.execute_prepared(
153
151
  :poll_jobs,
@@ -159,7 +157,7 @@ module Que
159
157
  )
160
158
 
161
159
  @last_polled_at = Time.now
162
- @last_poll_satisfied = expected_count == jobs.count
160
+ @last_poll_satisfied = poll_satisfied?(priorities, jobs)
163
161
 
164
162
  Que.internal_log :poller_polled, self do
165
163
  {
@@ -265,5 +263,12 @@ module Que
265
263
  SQL
266
264
  end
267
265
  end
266
+
267
+ private
268
+
269
+ def poll_satisfied?(priorities, jobs)
270
+ lowest_priority = priorities.keys.max
271
+ jobs.count >= priorities[lowest_priority]
272
+ end
268
273
  end
269
274
  end
@@ -40,8 +40,11 @@ module Que
40
40
  where(QUALIFIED_TABLE[:data].pg_jsonb.contains(JSON.dump(tags: [tag])))
41
41
  end
42
42
 
43
- def by_args(*args)
44
- where(QUALIFIED_TABLE[:args].pg_jsonb.contains(JSON.dump(args)))
43
+ def by_args(*args, **kwargs)
44
+ where(
45
+ QUALIFIED_TABLE[:args].pg_jsonb.contains(JSON.dump(args)) &
46
+ QUALIFIED_TABLE[:kwargs].pg_jsonb.contains(JSON.dump(kwargs))
47
+ )
45
48
  end
46
49
  end
47
50
  end
@@ -0,0 +1,19 @@
1
+ # frozen_string_literal: true
2
+
3
+ # Temporary module allowing ruby2 keyword args to be extracted from an *args splat
4
+ # Allows us to ensure consistent behaviour when running on ruby 2 vs ruby 3
5
+ # We can remove this if/when we drop support for ruby 2
6
+
7
+ require 'json'
8
+
9
+ module Que
10
+ module Utils
11
+ module Ruby2Keywords
12
+ def split_out_ruby2_keywords(args)
13
+ return [args, {}] unless args.last&.is_a?(Hash) && Hash.ruby2_keywords_hash?(args.last)
14
+
15
+ [args[0..-2], args.last]
16
+ end
17
+ end
18
+ end
19
+ end
data/lib/que/version.rb CHANGED
@@ -1,9 +1,9 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Que
4
- VERSION = '1.3.0'
4
+ VERSION = '2.0.0.beta1'
5
5
 
6
6
  def self.job_schema_version
7
- 1
7
+ 2
8
8
  end
9
9
  end
data/lib/que/worker.rb CHANGED
@@ -125,7 +125,7 @@ module Que
125
125
  log_message[:event] = :job_worked
126
126
  end
127
127
 
128
- Que.log(log_message)
128
+ Que.log(**log_message)
129
129
  end
130
130
 
131
131
  instance
data/lib/que.rb CHANGED
@@ -29,8 +29,11 @@ module Que
29
29
  require_relative 'que/utils/logging'
30
30
  require_relative 'que/utils/middleware'
31
31
  require_relative 'que/utils/queue_management'
32
+ require_relative 'que/utils/ruby2_keywords'
32
33
  require_relative 'que/utils/transactions'
33
34
 
35
+ require_relative 'que/version'
36
+
34
37
  require_relative 'que/connection'
35
38
  require_relative 'que/connection_pool'
36
39
  require_relative 'que/job_methods'
@@ -41,7 +44,6 @@ module Que
41
44
  require_relative 'que/migrations'
42
45
  require_relative 'que/poller'
43
46
  require_relative 'que/result_queue'
44
- require_relative 'que/version'
45
47
  require_relative 'que/worker'
46
48
 
47
49
  class << self
@@ -60,6 +62,7 @@ module Que
60
62
  include Utils::Logging
61
63
  include Utils::Middleware
62
64
  include Utils::QueueManagement
65
+ include Utils::Ruby2Keywords
63
66
  include Utils::Transactions
64
67
 
65
68
  extend Forwardable
data/que.gemspec CHANGED
@@ -13,6 +13,8 @@ Gem::Specification.new do |spec|
13
13
  spec.homepage = 'https://github.com/que-rb/que'
14
14
  spec.license = 'MIT'
15
15
 
16
+ spec.required_ruby_version = '>= 2.7.0'
17
+
16
18
  files_to_exclude = [
17
19
  /\A\.circleci/,
18
20
  /\AGemfile/,