que 2.0.0.beta1 → 2.0.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 382ef488f1aa4e91e0ef2b617ad03bdfee4001020a71ef31662cd99e52e34915
4
- data.tar.gz: 1d59264e766fd94aeeec463d7ea7f4cf76c867bc356df553f7d9e7a705095da6
3
+ metadata.gz: c3e59fff6666e5a56dca1724d50cc27d6a0afc1e74c7cb7441ff21363a2cfc1a
4
+ data.tar.gz: ce13aa6c7b0204ca1e308d7355698f6be752aed5e7303e2eaf14d34780b21ff1
5
5
  SHA512:
6
- metadata.gz: 03b72732163fc9d24fe91a5dcbc18986c7dead94fe0a8d28e66727cf93739e9a47c4ab9f39239ff3776b63f24339c2266becc2b00bd1d45efc15f3a064580466
7
- data.tar.gz: 87d9701463b8e3ac5a7ba62cd376e4508a856de0c69b04e7dcdaf7d481746552d4d4762026b70efe4a910b6a7de9243dc98eeb6e43d4e7fbc3d1065b0b6a941e
6
+ metadata.gz: 855c912842adb2097d583a7dc17dd5744b1283ed17e9e21dacc3ebaee238d8d3bc1e1b89e95cf4843cb173d1eb5def298596a53986ee37abaf41f6ab34a109c7
7
+ data.tar.gz: '09a6a17641f976b708d9c8188c68d70f098faace18de5713df48e6715c4bc2f657a5d83c58773e43d802f531f328b57b604539beb81b18ea22c787a5f116b8a7'
data/CHANGELOG.md CHANGED
@@ -2,6 +2,8 @@
2
2
 
3
3
  <!-- MarkdownTOC autolink=true -->
4
4
 
5
+ - [2.0.0 \(2022-08-25\)](#200-2022-08-25)
6
+ - [1.4.1 \(2022-07-24\)](#141-2022-07-24)
5
7
  - [2.0.0.beta1 \(2022-03-24\)](#200beta1-2022-03-24)
6
8
  - [1.4.0 \(2022-03-23\)](#140-2022-03-23)
7
9
  - [1.3.1 \(2022-02-25\)](#131-2022-02-25)
@@ -52,6 +54,20 @@
52
54
 
53
55
  <!-- /MarkdownTOC -->
54
56
 
57
+ ## 2.0.0 (2022-08-25)
58
+
59
+ See beta 2.0.0.beta1, plus:
60
+
61
+ - **Fixed**:
62
+ + Updated to use non-deprecated method from PG when params are passed (`#async_exec_params`). [#374](https://github.com/que-rb/que/pull/374)
63
+
64
+ Note that @dtcristo submitted a PR proposing an easier upgrade path to Que 2 and Ruby 3 - [#365](https://github.com/que-rb/que/pull/365). We are yet to properly consider this, but a later release which includes this feature would mean you don't need to simultaneously deploy Que 1.x and 2.x workers during the upgrade.
65
+
66
+ ## 1.4.1 (2022-07-24)
67
+
68
+ - **Added**
69
+ + Added Ruby version requirement of < 3. For Ruby 3 compatibility, upgrade to Que 2 - [upgrade process](https://github.com/que-rb/que/blob/master/CHANGELOG.md#200beta1-2022-03-24)
70
+
55
71
  ## 2.0.0.beta1 (2022-03-24)
56
72
 
57
73
  **Preliminary release of Ruby 3 support**
@@ -105,6 +121,8 @@ To run workers with two different versions of Que, you'll probably need to tempo
105
121
  - Update the bundle at `que-1-gemfile/Gemfile.lock` using `BUNDLE_GEMFILE=que-1-gemfile/Gemfile bundle`
106
122
  - Create a second deployment of Que, but with your `que` command prefixed with `BUNDLE_GEMFILE=que-1-gemfile/Gemfile`
107
123
 
124
+ We'd appreciate feedback on your experience upgrading to and running Que 2. Feel free to post on our Discord, or if you run into trouble, open an issue on GitHub.
125
+
108
126
  ## 1.4.0 (2022-03-23)
109
127
 
110
128
  - **Fixed**
@@ -115,7 +133,7 @@ To run workers with two different versions of Que, you'll probably need to tempo
115
133
  * It became used in 1.0.0.beta4, and that changelog entry has been updated to reflect this.
116
134
  - **Documentation**:
117
135
  + Reformatted the changelog to be more consistent, including adding links to all issue/PR numbers. [#347](https://github.com/que-rb/que/pull/347)
118
- +
136
+
119
137
  ## 1.3.1 (2022-02-25)
120
138
 
121
139
  Unfortunately, v1.3.0 was broken. Follow its upgrade instructions, but use this version instead.
data/README.md CHANGED
@@ -23,7 +23,7 @@ Que's secondary goal is performance. The worker process is multithreaded, so tha
23
23
 
24
24
  Compatibility:
25
25
 
26
- - MRI Ruby 2.7+
26
+ - MRI Ruby 2.7+ (for Ruby 3, Que 2+ is required)
27
27
  - PostgreSQL 9.5+
28
28
  - Rails 6.0+ (optional)
29
29
 
@@ -119,7 +119,11 @@ You can also add options to run the job after a specific time, or with a specifi
119
119
  ```ruby
120
120
  ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 })
121
121
  ```
122
+
123
+ [Learn more about job options](docs/README.md#job-options).
124
+
122
125
  ## Running the Que Worker
126
+
123
127
  In order to process jobs, you must start a separate worker process outside of your main server.
124
128
 
125
129
  ```bash
@@ -192,11 +196,11 @@ If you have a project that uses or relates to Que, feel free to submit a PR addi
192
196
 
193
197
  ## Community and Contributing
194
198
 
195
- - For bugs in the library, please feel free to [open an issue](https://github.com/que-rb/que/issues/new).
199
+ - For feature suggestions or bugs in the library, please feel free to [open an issue](https://github.com/que-rb/que/issues/new).
196
200
  - For general discussion and questions/concerns that don't relate to obvious bugs, join our [Discord Server](https://discord.gg/B3EW32H).
197
201
  - For contributions, pull requests submitted via Github are welcome.
198
202
 
199
- Regarding contributions, one of the project's priorities is to keep Que as simple, lightweight and dependency-free as possible, and pull requests that change too much or wouldn't be useful to the majority of Que's users have a good chance of being rejected. If you're thinking of submitting a pull request that adds a new feature, consider starting a discussion in [que-talk](https://groups.google.com/forum/#!forum/que-talk) first about what it would do and how it would be implemented. If it's a sufficiently large feature, or if most of Que's users wouldn't find it useful, it may be best implemented as a standalone gem, like some of the related projects above.
203
+ Regarding contributions, one of the project's priorities is to keep Que as simple, lightweight and dependency-free as possible, and pull requests that change too much or wouldn't be useful to the majority of Que's users have a good chance of being rejected. If you're thinking of submitting a pull request that adds a new feature, consider starting a discussion in an issue first about what it would do and how it would be implemented. If it's a sufficiently large feature, or if most of Que's users wouldn't find it useful, it may be best implemented as a standalone gem, like some of the related projects above.
200
204
 
201
205
  ### Specs
202
206
 
data/docs/README.md CHANGED
@@ -1,52 +1,62 @@
1
- Docs Index
2
- ===============
1
+ # Que documentation
2
+
3
+ <!-- MarkdownTOC autolink=true -->
3
4
 
4
5
  - [Command Line Interface](#command-line-interface)
5
- * [worker-priorities and worker-count](#worker-priorities-and-worker-count)
6
- * [poll-interval](#poll-interval)
7
- * [maximum-buffer-size](#maximum-buffer-size)
8
- * [connection-url](#connection-url)
9
- * [wait-period](#wait-period)
10
- * [log-internals](#log-internals)
6
+ - [`worker-priorities` and `worker-count`](#worker-priorities-and-worker-count)
7
+ - [`poll-interval`](#poll-interval)
8
+ - [`maximum-buffer-size`](#maximum-buffer-size)
9
+ - [`connection-url`](#connection-url)
10
+ - [`wait-period`](#wait-period)
11
+ - [`log-internals`](#log-internals)
11
12
  - [Advanced Setup](#advanced-setup)
12
- * [Using ActiveRecord Without Rails](#using-activerecord-without-rails)
13
- * [Managing the Jobs Table](#managing-the-jobs-table)
14
- * [Other Setup](#other-setup)
13
+ - [Using ActiveRecord Without Rails](#using-activerecord-without-rails)
14
+ - [Managing the Jobs Table](#managing-the-jobs-table)
15
+ - [Other Setup](#other-setup)
15
16
  - [Error Handling](#error-handling)
16
- * [Error Notifications](#error-notifications)
17
- * [Error-Specific Handling](#error-specific-handling)
17
+ - [Error Notifications](#error-notifications)
18
+ - [Error-Specific Handling](#error-specific-handling)
18
19
  - [Inspecting the Queue](#inspecting-the-queue)
19
- * [Job Stats](#job-stats)
20
- * [Custom Queries](#custom-queries)
21
- + [ActiveRecord Example](#activerecord-example)
22
- + [Sequel Example](#sequel-example)
20
+ - [Job Stats](#job-stats)
21
+ - [Custom Queries](#custom-queries)
22
+ - [ActiveRecord Example](#activerecord-example)
23
+ - [Sequel Example](#sequel-example)
23
24
  - [Managing Workers](#managing-workers)
24
- * [Working Jobs Via Executable](#working-jobs-via-executable)
25
- * [Thread-Unsafe Application Code](#thread-unsafe-application-code)
25
+ - [Working Jobs Via Executable](#working-jobs-via-executable)
26
+ - [Thread-Unsafe Application Code](#thread-unsafe-application-code)
26
27
  - [Logging](#logging)
27
- * [Logging Job Completion](#logging-job-completion)
28
+ - [Logging Job Completion](#logging-job-completion)
28
29
  - [Migrating](#migrating)
29
30
  - [Multiple Queues](#multiple-queues)
30
31
  - [Shutting Down Safely](#shutting-down-safely)
31
32
  - [Using Plain Postgres Connections](#using-plain-postgres-connections)
32
- * [Using ConnectionPool or Pond](#using-connectionpool-or-pond)
33
- * [Using Any Other Connection Pool](#using-any-other-connection-pool)
33
+ - [Using ConnectionPool or Pond](#using-connectionpool-or-pond)
34
+ - [Using Any Other Connection Pool](#using-any-other-connection-pool)
34
35
  - [Using Sequel](#using-sequel)
35
36
  - [Using Que With ActiveJob](#using-que-with-activejob)
36
37
  - [Job Helper Methods](#job-helper-methods)
37
- * [destroy](#destroy)
38
- * [finish](#finish)
39
- * [expire](#expire)
40
- * [retry_in](#retry_in)
41
- * [error_count](#error_count)
42
- * [default_resolve_action](#default_resolve_action)
38
+ - [`destroy`](#destroy)
39
+ - [`finish`](#finish)
40
+ - [`expire`](#expire)
41
+ - [`retry_in`](#retry_in)
42
+ - [`error_count`](#error_count)
43
+ - [`default_resolve_action`](#default_resolve_action)
43
44
  - [Writing Reliable Jobs](#writing-reliable-jobs)
44
- * [Timeouts](#timeouts)
45
+ - [Timeouts](#timeouts)
46
+ - [Job Options](#job-options)
47
+ - [`queue`](#queue)
48
+ - [`priority`](#priority)
49
+ - [`run_at`](#run_at)
50
+ - [`job_class`](#job_class)
51
+ - [`tags`](#tags)
45
52
  - [Middleware](#middleware)
46
- * [Defining Middleware For Jobs](#defining-middleware-for-jobs)
47
- * [Defining Middleware For SQL statements](#defining-middleware-for-sql-statements)
53
+ - [Defining Middleware For Jobs](#defining-middleware-for-jobs)
54
+ - [Defining Middleware For SQL statements](#defining-middleware-for-sql-statements)
48
55
  - [Vacuuming](#vacuuming)
56
+ - [Expired jobs](#expired-jobs)
57
+ - [Finished jobs](#finished-jobs)
49
58
 
59
+ <!-- /MarkdownTOC -->
50
60
 
51
61
  ## Command Line Interface
52
62
 
@@ -67,7 +77,7 @@ usage: que [options] [file/to/require] ...
67
77
 
68
78
  Some explanation of the more unusual options:
69
79
 
70
- ### worker-priorities and worker-count
80
+ ### `worker-priorities` and `worker-count`
71
81
 
72
82
  These options dictate the size and priority distribution of the worker pool. The default worker-priorities is `10,30,50,any,any,any`. This means that the default worker pool will reserve one worker to only works jobs with priorities under 10, one for priorities under 30, and one for priorities under 50. Three more workers will work any job.
73
83
 
@@ -77,23 +87,23 @@ Instead of passing worker-priorities, you can pass a `worker-count` - this is a
77
87
 
78
88
  If you pass both worker-count and worker-priorities, the count will trim or pad the priorities list with `any` workers. So, `--worker-priorities=20,30,40 --worker-count=6` would be the same as passing `--worker-priorities=20,30,40,any,any,any`.
79
89
 
80
- ### poll-interval
90
+ ### `poll-interval`
81
91
 
82
92
  This option sets the number of seconds the process will wait between polls of the job queue. Jobs that are ready to be worked immediately will be broadcast via the LISTEN/NOTIFY system, so polling is unnecessary for them - polling is only necessary for jobs that are scheduled in the future or which are being delayed due to errors. The default is 5 seconds.
83
93
 
84
- ### maximum-buffer-size
94
+ ### `maximum-buffer-size`
85
95
 
86
96
  This option sets the size of the internal buffer that Que uses to hold jobs until they're ready for workers. The default maximum is 8, meaning that the process won't buffer more than 8 jobs that aren't yet ready to be worked. If you don't want jobs to be buffered at all, you can set this value to zero.
87
97
 
88
- ### connection-url
98
+ ### `connection-url`
89
99
 
90
100
  This option sets the URL to be used to open a connection to the database for locking purposes. By default, Que will simply use a connection from the connection pool for locking - this option is only useful if your application connections can't use advisory locks - for example, if they're passed through an external connection pool like PgBouncer. In that case, you'll need to use this option to specify your actual database URL so that Que can establish a direct connection.
91
101
 
92
- ### wait-period
102
+ ### `wait-period`
93
103
 
94
104
  This option specifies (in milliseconds) how often the locking thread wakes up to check whether the workers have finished jobs, whether it's time to poll, etc. You shouldn't generally need to tweak this, but it may come in handy for some workloads. The default is 50 milliseconds.
95
105
 
96
- ### log-internals
106
+ ### `log-internals`
97
107
 
98
108
  This option instructs Que to output a lot of information about its internal state to the logger. It should only be used if it becomes necessary to debug issues.
99
109
 
@@ -147,7 +157,6 @@ Be sure to read the docs on [managing workers](#managing-workers) for more infor
147
157
 
148
158
  You'll also want to set up [logging](#logging) and an [error handler](#error-handling) to track errors raised by jobs.
149
159
 
150
-
151
160
  ## Error Handling
152
161
 
153
162
  If an error is raised and left uncaught by your job, Que will save the error message and backtrace to the database and schedule the job to be retried later.
@@ -425,7 +434,6 @@ Que.db_version #=> 3
425
434
 
426
435
  Note that you can remove Que from your database completely by migrating to version 0.
427
436
 
428
-
429
437
  ## Multiple Queues
430
438
 
431
439
  Que supports the use of multiple queues in a single job table. Please note that this feature is intended to support the case where multiple codebases are sharing the same job queue - if you want to support jobs of differing priorities, the numeric priority system offers better flexibility and performance.
@@ -452,11 +460,7 @@ class ProcessCreditCard < Que::Job
452
460
  end
453
461
  ```
454
462
 
455
- In some cases, the ProcessCreditCard class may not be defined in the application that is enqueueing the job. In that case, you can specify the job class as a string:
456
-
457
- ```ruby
458
- Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard', queue: 'credit_cards' })
459
- ```
463
+ In some cases, the `ProcessCreditCard` class may not be defined in the application that is enqueueing the job. In that case, you can [specify the job class as a string](#job_class).
460
464
 
461
465
  ## Shutting Down Safely
462
466
 
@@ -466,7 +470,6 @@ To prevent this, Que will block the worker process from exiting until all jobs i
466
470
 
467
471
  So, be prepared to use SIGKILL on your Ruby processes if they run for too long. For example, Heroku takes a good approach to this - when Heroku's platform is shutting down a process, it sends SIGTERM, waits ten seconds, then sends SIGKILL if the process still hasn't exited. This is a nice compromise - it will give each of your currently running jobs ten seconds to complete, and any jobs that haven't finished by then will be interrupted and retried later.
468
472
 
469
-
470
473
  ## Using Plain Postgres Connections
471
474
 
472
475
  If you're not using an ORM like ActiveRecord or Sequel, you can use a distinct connection pool to manage your Postgres connections. Please be aware that if you **are** using ActiveRecord or Sequel, there's no reason for you to be using any of these methods - it's less efficient (unnecessary connections will waste memory on your database server) and you lose the reliability benefits of wrapping jobs in the same transactions as the rest of your data.
@@ -594,27 +597,29 @@ Additionally, including `Que::ActiveJob::JobExtensions` lets you define a run()
594
597
 
595
598
  There are a number of instance methods on Que::Job that you can use in your jobs, preferably in transactions. See [Writing Reliable Jobs](#writing-reliable-jobs) for more information on where to use these methods.
596
599
 
597
- ### destroy
600
+ ### `destroy`
598
601
 
599
602
  This method deletes the job from the queue table, ensuring that it won't be worked a second time.
600
603
 
601
- ### finish
604
+ ### `finish`
602
605
 
603
606
  This method marks the current job as finished, ensuring that it won't be worked a second time. This is like destroy, in that it finalizes a job, but this method leaves the job in the table, in case you want to query it later.
604
607
 
605
- ### expire
608
+ ### `expire`
606
609
 
607
610
  This method marks the current job as expired. It will be left in the table and won't be retried, but it will be easy to query for expired jobs. This method is called if the job exceeds its maximum_retry_count.
608
611
 
609
- ### retry_in
612
+ ### `retry_in`
610
613
 
611
614
  This method marks the current job to be retried later. You can pass a numeric to this method, in which case that is the number of seconds after which it can be retried (`retry_in(10)`, `retry_in(0.5)`), or, if you're using ActiveSupport, you can pass in a duration object (`retry_in(10.minutes)`). This automatically happens, with an exponentially-increasing interval, when the job encounters an error.
612
615
 
613
- ### error_count
616
+ Note that `retry_in` increments the job's `error_count`.
617
+
618
+ ### `error_count`
614
619
 
615
620
  This method returns the total number of times the job has errored, in case you want to modify the job's behavior after it has failed a given number of times.
616
621
 
617
- ### default_resolve_action
622
+ ### `default_resolve_action`
618
623
 
619
624
  If you don't perform a resolve action (destroy, finish, expire, retry_in) while the job is worked, Que will call this method for you. By default it simply calls `destroy`, but you can override it in your Job subclasses if you wish - for example, to call `finish`, or to invoke some more complicated logic.
620
625
 
@@ -727,6 +732,51 @@ end
727
732
 
728
733
  Now, if the request takes more than five seconds, an error will be raised (probably - check your library's documentation) and Que will just retry the job later.
729
734
 
735
+ ## Job Options
736
+
737
+ When enqueueing a job, you can specify particular options for it in a `job_options` hash, e.g.:
738
+
739
+ ```ruby
740
+ ChargeCreditCard.enqueue(card.id, user_id: current_user.id, job_options: { run_at: 1.day.from_now, priority: 5 })
741
+ ```
742
+
743
+ ### `queue`
744
+
745
+ See [Multiple Queues](#multiple-queues).
746
+
747
+ ### `priority`
748
+
749
+ Provide an integer to customise the priority level of the job.
750
+
751
+ We use the Linux priority scale - a lower number is more important.
752
+
753
+ ### `run_at`
754
+
755
+ Provide a `Time` as the `run_at` to make a job run at a later time (well, at some point after it, depending on how busy the workers are).
756
+
757
+ It's best not to use `Time.now` here, as the current time in the Ruby process and the database won't be perfectly aligned. When the database considers the `run_at` to be in the past, the job will not be broadcast via the LISTEN/NOTIFY system, and it will need to wait for a poll. This introduces an unnecessary delay of probably a few seconds (depending on your configured [poll interval](#poll-interval)). So if you want the job to run ASAP, just omit the `run_at` option.
758
+
759
+ ### `job_class`
760
+
761
+ Specifying `job_class` allows you to enqueue a job using `Que.enqueue`:
762
+
763
+ ```ruby
764
+ Que.enqueue(current_user.id, job_options: { job_class: 'ProcessCreditCard' })
765
+ ```
766
+
767
+ Rather than needing to use the job class (nor even have it defined in the enqueueing process):
768
+
769
+ ```ruby
770
+ ProcessCreditCard.enqueue(current_user.id)
771
+ ```
772
+
773
+ ### `tags`
774
+
775
+ You can provide an array of strings to give a job some tags. These are not used by Que and are completely custom.
776
+
777
+ A job can have up to five tags, each one up to 100 characters long.
778
+
779
+ Note that unlike the other job options, tags are stored within the `que_jobs.data` column, rather than a correspondingly-named column.
730
780
 
731
781
  ## Middleware
732
782
 
@@ -785,3 +835,23 @@ class ManualVacuumJob < CronJob
785
835
  end
786
836
  end
787
837
  ```
838
+
839
+ ## Expired jobs
840
+
841
+ Expired jobs hang around in the `que_jobs` table. If necessary, you can get an expired job to run again by clearing the `error_count` and `expired_at` columns, e.g.:
842
+
843
+ ```sql
844
+ UPDATE que_jobs SET error_count = 0, expired_at = NULL WHERE id = 172340879;
845
+ ```
846
+
847
+ ## Finished jobs
848
+
849
+ If you prefer to leave finished jobs in the database for a while, to performantly remove them periodically, you can use something like:
850
+
851
+ ```sql
852
+ BEGIN;
853
+ ALTER TABLE que_jobs DISABLE TRIGGER que_state_notify;
854
+ DELETE FROM que_jobs WHERE finished_at < (select now() - interval '7 days');
855
+ ALTER TABLE que_jobs ENABLE TRIGGER que_state_notify;
856
+ COMMIT;
857
+ ```
@@ -62,7 +62,7 @@ module Que
62
62
  if params.empty?
63
63
  wrapped_connection.async_exec(sql)
64
64
  else
65
- wrapped_connection.async_exec(sql, params)
65
+ wrapped_connection.async_exec_params(sql, params)
66
66
  end
67
67
  end
68
68
 
data/lib/que/version.rb CHANGED
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Que
4
- VERSION = '2.0.0.beta1'
4
+ VERSION = '2.0.0'
5
5
 
6
6
  def self.job_schema_version
7
7
  2
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: que
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.0.0.beta1
4
+ version: 2.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Chris Hanks
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2022-03-24 00:00:00.000000000 Z
11
+ date: 2022-08-25 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
@@ -109,9 +109,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
109
109
  version: 2.7.0
110
110
  required_rubygems_version: !ruby/object:Gem::Requirement
111
111
  requirements:
112
- - - ">"
112
+ - - ">="
113
113
  - !ruby/object:Gem::Version
114
- version: 1.3.1
114
+ version: '0'
115
115
  requirements: []
116
116
  rubygems_version: 3.3.7
117
117
  signing_key: