que 1.0.0.beta5 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +250 -1
- data/Dockerfile +20 -0
- data/README.md +86 -35
- data/auto/dev +21 -0
- data/auto/psql +9 -0
- data/auto/test +5 -0
- data/auto/test-postgres-14 +17 -0
- data/docker-compose.yml +46 -0
- data/lib/que/job.rb +32 -15
- data/lib/que/version.rb +1 -1
- data/lib/que/worker.rb +2 -1
- data/scripts/docker-entrypoint +14 -0
- data/scripts/test +5 -0
- metadata +15 -8
- data/CHANGELOG.1.0.beta.md +0 -137
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 967eded27a81df945b65911275dcdc7925113727bfd06fbc42cb780a3bfb6fcf
|
4
|
+
data.tar.gz: 87607b262263b13623b4b09822a478c02179e57cfa503f253e3e678ac8b1490a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3d8242fe07445a6f4658f322dd1a7cdb8918c8a56c2bfad2db49c58695b4ebae183e6a56159abc4ff74316006e7d95459ab6cfd8a4c98410eea50a2e4fb52509
|
7
|
+
data.tar.gz: 24fdf63cab7ff6e8a2ffa6ed598718d3c2ae2cbc6ef61de22ab6f1541b0cc23bab980fe5d10bbcb54823b2a529cd1cdc9d57437db3905e802bec92a04ea3ed94
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,71 @@
|
|
1
|
+
### 1.2.0 (2022-02-23)
|
2
|
+
|
3
|
+
- **Deprecated**
|
4
|
+
+ Providing job options as top level keyword arguments to Job.enqueue is now deprecated. Support will be dropped in `2.0`. Job options should be nested under the `job_options` keyword arg instead. See [#336](https://github.com/que-rb/que/pull/336)
|
5
|
+
|
6
|
+
### 1.1.0 (2022-02-21)
|
7
|
+
|
8
|
+
- **Features**:
|
9
|
+
+ Add backtrace to errors, by [@trammel](https://github.com/trammel) in [#328](https://github.com/que-rb/que/pull/328)
|
10
|
+
- **Internal**:
|
11
|
+
+ Add Dockerised development environment, in [#324](https://github.com/que-rb/que/pull/324)
|
12
|
+
|
13
|
+
### 1.0.0 (2022-01-24)
|
14
|
+
|
15
|
+
_This release does not add any changes on top of 1.0.0.beta5._
|
16
|
+
|
17
|
+
### 1.0.0.beta5 (2021-12-23)
|
18
|
+
|
19
|
+
* **Bug fixes and improvements**
|
20
|
+
* Add more context to error message when config files fail to load. by [@trammel](https://github.com/trammel) in [#293](https://github.com/que-rb/que/pull/293)
|
21
|
+
* Fix lock leak on PostgreSQL 12 and later by [@jasoncodes](https://github.com/jasoncodes) in [#298](https://github.com/que-rb/que/pull/298)
|
22
|
+
* Fix deadlock issue [#318](https://github.com/que-rb/que/pull/318)
|
23
|
+
* Fix thread attrition issue [#321](https://github.com/que-rb/que/pull/321)
|
24
|
+
|
25
|
+
* **Rails fixes:**
|
26
|
+
* Set schema in table_name for ActiveRecord model by [@nikitug](https://github.com/nikitug) in [#274](https://github.com/que-rb/que/pull/274)
|
27
|
+
|
28
|
+
* **Documentation:**
|
29
|
+
* Add link to que-locks for exclusive job locking by [@airhorns](https://github.com/airhorns) in [#263](https://github.com/que-rb/que/pull/263)
|
30
|
+
[`5259303`](https://github.com/que-rb/que/commit/52593031a7eef2d52ac38eceb2d8df776ec74090)
|
31
|
+
* Fix links to Writing Reliable Jobs by [@nikitug](https://github.com/nikitug) in [#273](https://github.com/que-rb/que/pull/273)
|
32
|
+
* Add build badge to README by [@jonathanhefner](https://github.com/jonathanhefner) in [#278](https://github.com/que-rb/que/pull/278)
|
33
|
+
* Fix ToC links in docs by [@swrobel](https://github.com/swrobel) in [#287](https://github.com/que-rb/que/pull/287)
|
34
|
+
* Note all Rails queue names that must be changed by [@swrobel](https://github.com/swrobel) in [#296](https://github.com/que-rb/que/pull/296)
|
35
|
+
* Add instructions for how to start Que by [@xcskier56](https://github.com/xcskier56) in [#292](https://github.com/que-rb/que/pull/292)
|
36
|
+
|
37
|
+
* **CI/tests**
|
38
|
+
* Fix CI failure from Docker Postgres image by [@jonathanhefner](https://github.com/jonathanhefner) in [#275](https://github.com/que-rb/que/pull/275)
|
39
|
+
* Test with Ruby 2.7 by [@jonathanhefner](https://github.com/jonathanhefner) in [#276](https://github.com/que-rb/que/pull/276)
|
40
|
+
* Run GitHub build workflow on push by [@jonathanhefner](https://github.com/jonathanhefner) in [#277](https://github.com/que-rb/que/pull/277)
|
41
|
+
|
42
|
+
|
43
|
+
**Full Changelog**: [`v1.0.0.beta4...v1.0.0.beta5`](https://github.com/que-rb/que/compare/v1.0.0.beta4...v1.0.0.beta5)
|
44
|
+
|
45
|
+
**Unless an issue is found we intend for this release to become v1.0.0 proper.**
|
46
|
+
|
47
|
+
---
|
48
|
+
|
49
|
+
### 1.0.0.beta4 (2020-01-17)
|
50
|
+
|
51
|
+
- Rails 6 compatibility: Fix time parsing #249 and https://github.com/que-rb/que/commit/5ddddd5ebac6153d7a683ef08c86bced8e03fb51
|
52
|
+
- Cleaner sequel usage #257
|
53
|
+
- Documentation improvements #264 #269 #261 #231
|
54
|
+
|
55
|
+
---
|
56
|
+
|
57
|
+
### 1.0.0.beta3 (2018-05-18)
|
58
|
+
|
59
|
+
* Added support for customizing log levels for `job_worked` events (#217).
|
60
|
+
|
61
|
+
* Began logging all `job_errored` events at the `ERROR` log level.
|
62
|
+
|
63
|
+
* Fixed the Railtie when running in test mode (#214).
|
64
|
+
|
65
|
+
* Tweaked the meanings of worker-priorities and worker-count options in the CLI, to better support use cases with low worker counts (#216).
|
66
|
+
|
67
|
+
---
|
68
|
+
|
1
69
|
### 1.0.0.beta2 (2018-04-13)
|
2
70
|
|
3
71
|
* **A schema upgrade to version 4 will be required for this release.** See [the migration doc](https://github.com/que-rb/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
|
@@ -111,8 +179,123 @@
|
|
111
179
|
* Names for queues and job classes are now limited to 500 characters, which is still far longer than either of these values should reasonably be.
|
112
180
|
|
113
181
|
* There is now a `data` JSONB column which is used to support various ways of organizing jobs (setting tags on them, etc).
|
182
|
+
---
|
183
|
+
|
184
|
+
### 1.0.0.beta (2017-10-25)
|
185
|
+
|
186
|
+
* **A schema upgrade to version 4 will be required for this release.** See [the migration doc](https://github.com/que-rb/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
|
187
|
+
|
188
|
+
* Please note that this migration requires a rewrite of the jobs table, which makes it O(n) with the size of the table. If you have a very large backlog of jobs you may want to schedule downtime for this migration.
|
189
|
+
|
190
|
+
* Que's implementation has been changed from one in which worker threads hold their own PG connections and lock their own jobs to one in which a single thread (and PG connection) locks jobs through LISTEN/NOTIFY and batch polling, and passes jobs along to worker threads. This has many benefits, including:
|
191
|
+
|
192
|
+
* Jobs queued for immediate processing can be actively distributed to workers with LISTEN/NOTIFY, which is more efficient than having workers repeatedly poll for new jobs.
|
193
|
+
|
194
|
+
* When polling is necessary (to pick up jobs that are scheduled for the future or that need to be retried due to errors), jobs can be locked and fetched in batches, rather than one at a time.
|
195
|
+
|
196
|
+
* Individual workers no longer need to monopolize their own (usually idle) connections while working jobs, so Ruby processes will require fewer Postgres connections.
|
197
|
+
|
198
|
+
* PgBouncer or another external connection pool can be used for workers' connections (though not for the connection used to lock and listen for jobs).
|
199
|
+
|
200
|
+
* Other features introduced in this version include:
|
201
|
+
|
202
|
+
* Much better support for all versions of ActiveJob.
|
114
203
|
|
115
|
-
|
204
|
+
* In particular, you may (optionally) include `Que::ActiveJob::JobExtensions` into `ApplicationJob` to get support for all of Que's job helper methods.
|
205
|
+
|
206
|
+
* Custom middleware that wrap running jobs are now supported.
|
207
|
+
|
208
|
+
* Support for categorizing jobs with tags.
|
209
|
+
|
210
|
+
* Support for configuring a `maximum_retry_count` on individual job classes.
|
211
|
+
|
212
|
+
* Job configuration options are now inheritable, so job class hierarchies are more useful.
|
213
|
+
|
214
|
+
* There are now built-in models for ActiveRecord and Sequel to allow inspecting the queue easily.
|
215
|
+
|
216
|
+
* Jobs that have finished working may optionally be retained in the database indefinitely.
|
217
|
+
|
218
|
+
* To keep a job record, replace the `destroy` calls in your jobs with `finish`. `destroy` will still delete records entirely, for jobs that you don't want to keep.
|
219
|
+
|
220
|
+
* If you don't resolve a job yourself one way or another, Que will still `destroy` the job for you by default.
|
221
|
+
|
222
|
+
* Finished jobs have a timestamp set in the finished_at column.
|
223
|
+
|
224
|
+
* Jobs that have errored too many times will now be marked as expired, and won't be retried again.
|
225
|
+
|
226
|
+
* You can configure a maximum_retry_count in your job classes, to set the threshold at which a job will be marked expired. The default is 15.
|
227
|
+
|
228
|
+
* To manually mark a job as expired (and keep it in the database but not try to run it again) you can call `expire` helper in your job.
|
229
|
+
|
230
|
+
* You can now set job priority thresholds for individual workers, to ensure that there will always be space available for high-priority jobs.
|
231
|
+
|
232
|
+
* `Que.job_states` returns a list of locked jobs and the hostname/pid of the Ruby processes that have locked them.
|
233
|
+
|
234
|
+
* `Que.connection_proc=` has been added, to allow for the easy integration of custom connection pools.
|
235
|
+
|
236
|
+
* In keeping with semantic versioning, the major version is being bumped since the new implementation requires some backwards-incompatible changes. These changes include:
|
237
|
+
|
238
|
+
* Support for MRI Rubies before 2.2 has been dropped.
|
239
|
+
|
240
|
+
* Support for Postgres versions before 9.5 has been dropped (JSONB and upsert support is required).
|
241
|
+
|
242
|
+
* JRuby support has been dropped. It will be reintroduced whenever the jruby-pg gem is production-ready.
|
243
|
+
|
244
|
+
* The `que:work` rake task has been removed. Use the `que` executable instead.
|
245
|
+
|
246
|
+
* Therefore, configuring workers using QUE_* environment variables is no longer supported. Please pass the appropriate options to the `que` executable instead.
|
247
|
+
|
248
|
+
* The `mode` setter has been removed.
|
249
|
+
|
250
|
+
* To run jobs synchronously when they are enqueued (the old `:sync` behavior) you can set `Que.run_synchronously = true`.
|
251
|
+
|
252
|
+
* To start up the worker pool (the old :async behavior) you should use the `que` executable to start up a worker process. There's no longer a supported API for running workers outside of the `que` executable.
|
253
|
+
|
254
|
+
* The following methods are not meaningful under the new implementation and have been removed:
|
255
|
+
|
256
|
+
* The `Que.wake_interval` getter and setter.
|
257
|
+
|
258
|
+
* The `Que.worker_count` getter and setter.
|
259
|
+
|
260
|
+
* `Que.wake!`
|
261
|
+
|
262
|
+
* `Que.wake_all!`
|
263
|
+
|
264
|
+
* Since Que needs a dedicated Postgres connection to manage job locks, running Que through a single PG connection is no longer supported.
|
265
|
+
|
266
|
+
* It's not clear that anyone ever actually did this.
|
267
|
+
|
268
|
+
* `Que.worker_states` has been removed, as the connection that locks a job is no longer the one that the job is using to run. Its functionality has been partially replaced with `Que.job_states`.
|
269
|
+
|
270
|
+
* When using Rails, for simplicity, job attributes and keys in argument hashes are now converted to symbols when retrieved from the database, rather than being converted to instances of HashWithIndifferentAccess.
|
271
|
+
|
272
|
+
* Arguments passed to jobs are now deep-frozen, to prevent unexpected behavior when the args are mutated and the job is reenqueued.
|
273
|
+
|
274
|
+
* Since JSONB is now used to store arguments, the order of argument hashes is no longer maintained.
|
275
|
+
|
276
|
+
* It wouldn't have been a good idea to rely on this anyway.
|
277
|
+
|
278
|
+
* Calling Que.log() directly is no longer supported/recommended.
|
279
|
+
|
280
|
+
* Features marked as deprecated in the final 0.x releases have been removed.
|
281
|
+
|
282
|
+
* Finally, if you've built up your own tooling and customizations around Que, you may need to be aware of some DB schema changes made in the migration to schema version #4.
|
283
|
+
|
284
|
+
* The `job_id` column has been renamed `id` and is now the primary key. This makes it easier to manage the queue using an ActiveRecord model.
|
285
|
+
|
286
|
+
* Finished jobs are now kept in the DB, unless you explicitly call `destroy`. If you want to query the DB for only jobs that haven't finished yet, add a `WHERE finished_at IS NULL` condition to your query, or use the not_finished scope on one of the provided ORM models.
|
287
|
+
|
288
|
+
* There is now an `expired_at` timestamp column, which is set when a job reaches its maximum number of retries and will not be attempted again.
|
289
|
+
|
290
|
+
* Due to popular demand, the default queue name is now "default" rather than an empty string. The migration will move pending jobs under the "" queue to the "default" queue.
|
291
|
+
|
292
|
+
* The `last_error` column has been split in two, to `last_error_message` and `last_error_backtrace`. These two columns are now limited to 500 and 10,000 characters, respectively. The migration will split old error data correctly, and truncate it if necessary.
|
293
|
+
|
294
|
+
* Names for queues and job classes are now limited to 500 characters, which is still far longer than either of these values should reasonably be.
|
295
|
+
|
296
|
+
* There is now a `data` JSONB column which is used to support various ways of organizing jobs (setting tags on them, etc).
|
297
|
+
|
298
|
+
---
|
116
299
|
|
117
300
|
### 0.14.3 (2018-03-02)
|
118
301
|
|
@@ -120,6 +303,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
120
303
|
|
121
304
|
* Recorded error messages are now truncated to the first 500 characters.
|
122
305
|
|
306
|
+
---
|
307
|
+
|
123
308
|
### 0.14.2 (2018-01-05)
|
124
309
|
|
125
310
|
* Deprecate the Que.disable_prepared_statements= accessors.
|
@@ -128,24 +313,34 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
128
313
|
|
129
314
|
* Update the generated Rails migration to declare a version. (NARKOZ)
|
130
315
|
|
316
|
+
---
|
317
|
+
|
131
318
|
### 0.14.1 (2017-12-14)
|
132
319
|
|
133
320
|
* Fix a bug with typecasting boolean values on Rails 5+.
|
134
321
|
|
322
|
+
---
|
323
|
+
|
135
324
|
### 0.14.0 (2017-08-11)
|
136
325
|
|
137
326
|
* Fix incompatibility with Rails 5.1.
|
138
327
|
|
139
328
|
* Drop support for waking an in-process worker when an ActiveRecord transaction commits.
|
140
329
|
|
330
|
+
---
|
331
|
+
|
141
332
|
### 0.13.1 (2017-07-05)
|
142
333
|
|
143
334
|
* Fix issue that caused error stacktraces to not be persisted in most cases.
|
144
335
|
|
336
|
+
---
|
337
|
+
|
145
338
|
### 0.13.0 (2017-06-08)
|
146
339
|
|
147
340
|
* Fix recurring JSON issues by dropping MultiJson support. Previously MultiJson was detected and used automatically, and now it's just ignored and stdlib JSON used instead, so this shouldn't require any code changes.
|
148
341
|
|
342
|
+
---
|
343
|
+
|
149
344
|
### 0.12.3 (2017-06-01)
|
150
345
|
|
151
346
|
* Fix incompatibility with MultiJson introduced by the previous release.
|
@@ -154,28 +349,40 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
154
349
|
|
155
350
|
* Fix security vulnerability in parsing JSON from the DB (by specifying create_additions: false). This shouldn't be a concern unless you were passing untrusted user input in your job arguments. (hmac)
|
156
351
|
|
352
|
+
---
|
353
|
+
|
157
354
|
### 0.12.1 (2017-01-22)
|
158
355
|
|
159
356
|
* Fix incompatibility with Rails 5.0. (#166) (nbibler, thedarkone)
|
160
357
|
|
358
|
+
---
|
359
|
+
|
161
360
|
### 0.12.0 (2016-09-09)
|
162
361
|
|
163
362
|
* The error_handler configuration option has been renamed to error_notifier, which is more descriptive of what it's actually supposed to do. You can still use error_handler for configuration, but you'll get a warning.
|
164
363
|
|
165
364
|
* Introduced a new framework for handling errors on a per-job basis. See the docs for more information. (#106, #147)
|
166
365
|
|
366
|
+
---
|
367
|
+
|
167
368
|
### 0.11.6 (2016-07-01)
|
168
369
|
|
169
370
|
* Fix for operating in nested transactions in Rails 5.0. (#160) (greysteil)
|
170
371
|
|
372
|
+
---
|
373
|
+
|
171
374
|
### 0.11.5 (2016-05-13)
|
172
375
|
|
173
376
|
* Fix error when running `que -v`. (#154) (hardbap)
|
174
377
|
|
378
|
+
---
|
379
|
+
|
175
380
|
### 0.11.4 (2016-03-03)
|
176
381
|
|
177
382
|
* Fix incompatibility with ActiveRecord 5.0.0.beta3. (#143, #144) (joevandyk)
|
178
383
|
|
384
|
+
---
|
385
|
+
|
179
386
|
### 0.11.3 (2016-02-26)
|
180
387
|
|
181
388
|
* Fixed bug with displaying the current version of the que executable. (#122) (hardbap)
|
@@ -186,14 +393,20 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
186
393
|
|
187
394
|
* String literals are now frozen on Ruby 2.3.
|
188
395
|
|
396
|
+
---
|
397
|
+
|
189
398
|
### 0.11.2 (2015-09-09)
|
190
399
|
|
191
400
|
* Fix Job class constantizing when ActiveSupport isn't loaded. (#121) (godfat)
|
192
401
|
|
402
|
+
---
|
403
|
+
|
193
404
|
### 0.11.1 (2015-09-04)
|
194
405
|
|
195
406
|
* The `rake que:work` rake task that was specific to Rails has been deprecated and will be removed in Que 1.0. A deprecation warning will display when it is run.
|
196
407
|
|
408
|
+
---
|
409
|
+
|
197
410
|
### 0.11.0 (2015-09-04)
|
198
411
|
|
199
412
|
* A command-line program has been added that can be used to work jobs in a more flexible manner than the previous rake task. Run `que -h` for more information.
|
@@ -208,6 +421,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
208
421
|
|
209
422
|
* If it exists, use String#constantize to constantize job classes, since ActiveSupport's constantize method behaves better with Rails' autoloading. (#115, #120) (joevandyk)
|
210
423
|
|
424
|
+
---
|
425
|
+
|
211
426
|
### 0.10.0 (2015-03-18)
|
212
427
|
|
213
428
|
* When working jobs via the rake task, Rails applications are now eager-loaded if present, to avoid problems with multithreading and autoloading. (#96) (hmarr)
|
@@ -216,28 +431,40 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
216
431
|
|
217
432
|
* Add Que.transaction() helper method, to aid in transaction management in migrations or when the user's ORM doesn't provide one. (#81)
|
218
433
|
|
434
|
+
---
|
435
|
+
|
219
436
|
### 0.9.2 (2015-02-05)
|
220
437
|
|
221
438
|
* Fix a bug wherein the at_exit hook in the railtie wasn't waiting for jobs to finish before exiting.
|
222
439
|
|
223
440
|
* Fix a bug wherein the que:work rake task wasn't waiting for jobs to finish before exiting. (#85) (tycooon)
|
224
441
|
|
442
|
+
---
|
443
|
+
|
225
444
|
### 0.9.1 (2015-01-11)
|
226
445
|
|
227
446
|
* Use now() rather than 'now' when inserting jobs, to avoid using an old value as the default run_at in prepared statements. (#74) (bgentry)
|
228
447
|
|
448
|
+
---
|
449
|
+
|
229
450
|
### 0.9.0 (2014-12-16)
|
230
451
|
|
231
452
|
* The error_handler callable is now passed two objects, the error and the job that raised it. If your current error_handler is a proc, as recommended in the docs, you shouldn't need to make any code changes, unless you want to use the job in your error handling. If your error_handler is a lambda, or another callable with a strict arity requirement, you'll want to change it before upgrading. (#69) (statianzo)
|
232
453
|
|
454
|
+
---
|
455
|
+
|
233
456
|
### 0.8.2 (2014-10-12)
|
234
457
|
|
235
458
|
* Fix errors raised during rollbacks in the ActiveRecord adapter, which remained silent until Rails 4.2. (#64, #65) (Strech)
|
236
459
|
|
460
|
+
---
|
461
|
+
|
237
462
|
### 0.8.1 (2014-07-28)
|
238
463
|
|
239
464
|
* Fix regression introduced in the `que:work` rake task by the `mode` / `worker_count` disentangling in 0.8.0. (#50)
|
240
465
|
|
466
|
+
---
|
467
|
+
|
241
468
|
### 0.8.0 (2014-07-12)
|
242
469
|
|
243
470
|
* A callable can now be set as the logger, like `Que.logger = proc { MyLogger.new }`. Que uses this in its Railtie for cleaner initialization, but it is also available for public use.
|
@@ -246,20 +473,28 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
246
473
|
|
247
474
|
* Fixed a similar bug wherein setting a wake_interval during application startup would break worker awakening after the process was forked.
|
248
475
|
|
476
|
+
---
|
477
|
+
|
249
478
|
### 0.7.3 (2014-05-19)
|
250
479
|
|
251
480
|
* When mode = :sync, don't touch the database at all when running jobs inline. Needed for ActiveJob compatibility (#46).
|
252
481
|
|
482
|
+
---
|
483
|
+
|
253
484
|
### 0.7.2 (2014-05-18)
|
254
485
|
|
255
486
|
* Fix issue wherein intermittent worker wakeups would not work after forking (#44).
|
256
487
|
|
488
|
+
---
|
489
|
+
|
257
490
|
### 0.7.1 (2014-04-29)
|
258
491
|
|
259
492
|
* Fix errors with prepared statements when ActiveRecord reconnects to the database. (dvrensk)
|
260
493
|
|
261
494
|
* Don't use prepared statements when inside a transaction. This negates the risk of a prepared statement error harming the entire transaction. The query that benefits the most from preparation is the job-lock CTE, which is never run in a transaction, so the performance impact should be negligible.
|
262
495
|
|
496
|
+
---
|
497
|
+
|
263
498
|
### 0.7.0 (2014-04-09)
|
264
499
|
|
265
500
|
* `JobClass.queue(*args)` has been deprecated and will be removed in version 1.0.0. Please use `JobClass.enqueue(*args)` instead.
|
@@ -270,6 +505,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
270
505
|
|
271
506
|
* The [Pond gem](https://github.com/chanks/pond) is now supported as a connection. It is very similar to the ConnectionPool gem, but creates connections lazily and is dynamically resizable.
|
272
507
|
|
508
|
+
---
|
509
|
+
|
273
510
|
### 0.6.0 (2014-02-04)
|
274
511
|
|
275
512
|
* **A schema upgrade to version 3 is required for this release.** See [the migration doc](https://github.com/que-rb/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
|
@@ -290,6 +527,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
290
527
|
|
291
528
|
* Job classes may now define their own logic for determining the retry interval when a job raises an error. See [error handling](https://github.com/que-rb/que/blob/master/docs/error_handling.md) for more information.
|
292
529
|
|
530
|
+
---
|
531
|
+
|
293
532
|
### 0.5.0 (2014-01-14)
|
294
533
|
|
295
534
|
* When running a worker pool inside your web process on ActiveRecord, Que will now wake a worker once a transaction containing a queued job is committed. (joevandyk, chanks)
|
@@ -312,6 +551,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
312
551
|
|
313
552
|
Now, when a process exits, if the worker pool is running (whether in a rake task or in a web process) the exit will be stalled until all workers have finished their current jobs. If you have long-running jobs, this may take a long time. If you need the process to exit immediately, you can SIGKILL without any threat of commiting prematurely.
|
314
553
|
|
554
|
+
---
|
555
|
+
|
315
556
|
### 0.4.0 (2014-01-05)
|
316
557
|
|
317
558
|
* Que.wake_all! was added, as a simple way to wake up all workers in the pool.
|
@@ -332,6 +573,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
332
573
|
|
333
574
|
* Much more internal cleanup.
|
334
575
|
|
576
|
+
---
|
577
|
+
|
335
578
|
### 0.3.0 (2013-12-21)
|
336
579
|
|
337
580
|
* Add Que.stop!, which immediately kills all jobs being worked in the process.
|
@@ -344,6 +587,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
344
587
|
|
345
588
|
* Clean up internals and hammer out several race conditions.
|
346
589
|
|
590
|
+
---
|
591
|
+
|
347
592
|
### 0.2.0 (2013-11-30)
|
348
593
|
|
349
594
|
* Officially support JRuby 1.7.5+. Earlier versions may work.
|
@@ -356,6 +601,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
356
601
|
|
357
602
|
* :sync mode now ignores scheduled jobs (jobs queued with a specific run_at).
|
358
603
|
|
604
|
+
---
|
605
|
+
|
359
606
|
### 0.1.0 (2013-11-18)
|
360
607
|
|
361
608
|
* Initial public release, after a test-driven rewrite.
|
@@ -366,6 +613,8 @@ For a detailed list of the changes between each beta release of 1.0.0, see [the
|
|
366
613
|
|
367
614
|
Added a Railtie for easier setup with Rails, as well as a migration generator.
|
368
615
|
|
616
|
+
---
|
617
|
+
|
369
618
|
### 0.0.1 (2013-11-07)
|
370
619
|
|
371
620
|
* Copy-pasted from an app of mine. Very Sequel-specific. Nobody look at it, let's pretend it never happened.
|
data/Dockerfile
ADDED
@@ -0,0 +1,20 @@
|
|
1
|
+
FROM ruby:2.7.5-slim-buster@sha256:4cbbe2fba099026b243200aa8663f56476950cc64ccd91d7aaccddca31e445b5 AS base
|
2
|
+
|
3
|
+
# Install libpq-dev in our base layer, as it's needed in all environments
|
4
|
+
RUN apt-get update \
|
5
|
+
&& apt-get install -y libpq-dev \
|
6
|
+
&& rm -rf /var/lib/apt/lists/*
|
7
|
+
|
8
|
+
ENV RUBY_BUNDLER_VERSION 2.3.7
|
9
|
+
RUN gem install bundler -v $RUBY_BUNDLER_VERSION
|
10
|
+
|
11
|
+
ENV BUNDLE_PATH /usr/local/bundle
|
12
|
+
|
13
|
+
ENV RUBYOPT=-W:deprecated
|
14
|
+
|
15
|
+
FROM base AS dev-environment
|
16
|
+
|
17
|
+
# Install build-essential and git, as we'd need them for building gems that have native code components
|
18
|
+
RUN apt-get update \
|
19
|
+
&& apt-get install -y build-essential git \
|
20
|
+
&& rm -rf /var/lib/apt/lists/*
|
data/README.md
CHANGED
@@ -1,54 +1,59 @@
|
|
1
1
|
# Que ![tests](https://github.com/que-rb/que/workflows/tests/badge.svg)
|
2
2
|
|
3
|
-
**This README and the rest of the docs on the master branch all refer to Que 1.0
|
3
|
+
**This README and the rest of the docs on the master branch all refer to Que 1.0. If you're using version 0.x, please refer to the docs on [the 0.x branch](https://github.com/que-rb/que/tree/0.x).**
|
4
4
|
|
5
5
|
*TL;DR: Que is a high-performance job queue that improves the reliability of your application by protecting your jobs with the same [ACID guarantees](https://en.wikipedia.org/wiki/ACID) as the rest of your data.*
|
6
6
|
|
7
7
|
Que ("keɪ", or "kay") is a queue for Ruby and PostgreSQL that manages jobs using [advisory locks](http://www.postgresql.org/docs/current/static/explicit-locking.html#ADVISORY-LOCKS), which gives it several advantages over other RDBMS-backed queues:
|
8
8
|
|
9
|
-
|
10
|
-
|
11
|
-
|
9
|
+
- **Concurrency** - Workers don't block each other when trying to lock jobs, as often occurs with "SELECT FOR UPDATE"-style locking. This allows for very high throughput with a large number of workers.
|
10
|
+
- **Efficiency** - Locks are held in memory, so locking a job doesn't incur a disk write. These first two points are what limit performance with other queues. Under heavy load, Que's bottleneck is CPU, not I/O.
|
11
|
+
- **Safety** - If a Ruby process dies, the jobs it's working won't be lost, or left in a locked or ambiguous state - they immediately become available for any other worker to pick up.
|
12
12
|
|
13
13
|
Additionally, there are the general benefits of storing jobs in Postgres, alongside the rest of your data, rather than in Redis or a dedicated queue:
|
14
14
|
|
15
|
-
|
16
|
-
|
17
|
-
|
18
|
-
|
15
|
+
- **Transactional Control** - Queue a job along with other changes to your database, and it'll commit or rollback with everything else. If you're using ActiveRecord or Sequel, Que can piggyback on their connections, so setup is simple and jobs are protected by the transactions you're already using.
|
16
|
+
- **Atomic Backups** - Your jobs and data can be backed up together and restored as a snapshot. If your jobs relate to your data (and they usually do), there's no risk of jobs falling through the cracks during a recovery.
|
17
|
+
- **Fewer Dependencies** - If you're already using Postgres (and you probably should be), a separate queue is another moving part that can break.
|
18
|
+
- **Security** - Postgres' support for SSL connections keeps your data safe in transport, for added protection when you're running workers on cloud platforms that you can't completely control.
|
19
19
|
|
20
20
|
Que's primary goal is reliability. You should be able to leave your application running indefinitely without worrying about jobs being lost due to a lack of transactional support, or left in limbo due to a crashing process. Que does everything it can to ensure that jobs you queue are performed exactly once (though the occasional repetition of a job can be impossible to avoid - see the docs on [how to write a reliable job](/docs/README.md#writing-reliable-jobs)).
|
21
21
|
|
22
22
|
Que's secondary goal is performance. The worker process is multithreaded, so that a single process can run many jobs simultaneously.
|
23
23
|
|
24
24
|
Compatibility:
|
25
|
+
|
25
26
|
- MRI Ruby 2.2+
|
26
27
|
- PostgreSQL 9.5+
|
27
28
|
- Rails 4.1+ (optional)
|
28
29
|
|
29
30
|
**Please note** - Que's job table undergoes a lot of churn when it is under high load, and like any heavily-written table, is susceptible to bloat and slowness if Postgres isn't able to clean it up. The most common cause of this is long-running transactions, so it's recommended to try to keep all transactions against the database housing Que's job table as short as possible. This is good advice to remember for any high-activity database, but bears emphasizing when using tables that undergo a lot of writes.
|
30
31
|
|
31
|
-
|
32
32
|
## Installation
|
33
33
|
|
34
34
|
Add this line to your application's Gemfile:
|
35
35
|
|
36
|
-
|
36
|
+
```ruby
|
37
|
+
gem 'que'
|
38
|
+
```
|
37
39
|
|
38
40
|
And then execute:
|
39
41
|
|
40
|
-
|
42
|
+
```bash
|
43
|
+
bundle
|
44
|
+
```
|
41
45
|
|
42
46
|
Or install it yourself as:
|
43
47
|
|
44
|
-
|
45
|
-
|
48
|
+
```bash
|
49
|
+
gem install que
|
50
|
+
```
|
46
51
|
|
47
52
|
## Usage
|
48
53
|
|
49
54
|
First, create the queue schema in a migration. For example:
|
50
55
|
|
51
|
-
```
|
56
|
+
```ruby
|
52
57
|
class CreateQueSchema < ActiveRecord::Migration[5.0]
|
53
58
|
def up
|
54
59
|
# Whenever you use Que in a migration, always specify the version you're
|
@@ -66,7 +71,7 @@ end
|
|
66
71
|
|
67
72
|
Create a class for each type of job you want to run:
|
68
73
|
|
69
|
-
```
|
74
|
+
```ruby
|
70
75
|
# app/jobs/charge_credit_card.rb
|
71
76
|
class ChargeCreditCard < Que::Job
|
72
77
|
# Default settings for this job. These are optional - without them, jobs
|
@@ -101,7 +106,7 @@ end
|
|
101
106
|
|
102
107
|
Queue your job. Again, it's best to do this in a transaction with other changes you're making. Also note that any arguments you pass will be serialized to JSON and back again, so stick to simple types (strings, integers, floats, hashes, and arrays).
|
103
108
|
|
104
|
-
```
|
109
|
+
```ruby
|
105
110
|
CreditCard.transaction do
|
106
111
|
# Persist credit card information
|
107
112
|
card = CreditCard.create(params[:credit_card])
|
@@ -111,17 +116,18 @@ end
|
|
111
116
|
|
112
117
|
You can also add options to run the job after a specific time, or with a specific priority:
|
113
118
|
|
114
|
-
```
|
119
|
+
```ruby
|
115
120
|
ChargeCreditCard.enqueue card.id, user_id: current_user.id, run_at: 1.day.from_now, priority: 5
|
116
121
|
```
|
117
122
|
## Running the Que Worker
|
118
123
|
In order to process jobs, you must start a separate worker process outside of your main server.
|
119
124
|
|
120
|
-
```
|
125
|
+
```bash
|
121
126
|
bundle exec que
|
122
127
|
```
|
123
128
|
|
124
129
|
Try running `que -h` to get a list of runtime options:
|
130
|
+
|
125
131
|
```
|
126
132
|
$ que -h
|
127
133
|
usage: que [options] [file/to/require] ...
|
@@ -138,21 +144,24 @@ If you're using ActiveRecord to dump your database's schema, please [set your sc
|
|
138
144
|
|
139
145
|
Pre-1.0, the default queue name needed to be configured in order for Que to work out of the box with Rails. In 1.0 the default queue name is now 'default', as Rails expects, but when Rails enqueues some types of jobs it may try to use another queue name that isn't worked by default. You can either:
|
140
146
|
|
141
|
-
|
142
|
-
```ruby
|
143
|
-
config.action_mailer.deliver_later_queue_name = :default
|
144
|
-
config.action_mailbox.queues.incineration = :default
|
145
|
-
config.action_mailbox.queues.routing = :default
|
146
|
-
config.active_storage.queues.analysis = :default
|
147
|
-
config.active_storage.queues.purge = :default
|
148
|
-
```
|
147
|
+
- [Configure Rails](https://guides.rubyonrails.org/configuring.html) to send all internal job types to the 'default' queue by adding the following to `config/application.rb`:
|
149
148
|
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
149
|
+
```ruby
|
150
|
+
config.action_mailer.deliver_later_queue_name = :default
|
151
|
+
config.action_mailbox.queues.incineration = :default
|
152
|
+
config.action_mailbox.queues.routing = :default
|
153
|
+
config.active_storage.queues.analysis = :default
|
154
|
+
config.active_storage.queues.purge = :default
|
155
|
+
```
|
156
|
+
|
157
|
+
- [Tell que](/docs#multiple-queues) to work all of these queues (less efficient because it requires polling all of them):
|
158
|
+
|
159
|
+
```bash
|
160
|
+
que -q default -q mailers -q action_mailbox_incineration -q action_mailbox_routing -q active_storage_analysis -q active_storage_purge
|
161
|
+
```
|
154
162
|
|
155
163
|
Also, if you would like to integrate Que with Active Job, you can do it by setting the adapter in `config/application.rb` or in a specific environment by setting it in `config/environments/production.rb`, for example:
|
164
|
+
|
156
165
|
```ruby
|
157
166
|
config.active_job.queue_adapter = :que
|
158
167
|
```
|
@@ -183,9 +192,9 @@ If you have a project that uses or relates to Que, feel free to submit a PR addi
|
|
183
192
|
|
184
193
|
## Community and Contributing
|
185
194
|
|
186
|
-
|
187
|
-
|
188
|
-
|
195
|
+
- For bugs in the library, please feel free to [open an issue](https://github.com/que-rb/que/issues/new).
|
196
|
+
- For general discussion and questions/concerns that don't relate to obvious bugs, join our [Discord Server](https://discord.gg/B3EW32H).
|
197
|
+
- For contributions, pull requests submitted via Github are welcome.
|
189
198
|
|
190
199
|
Regarding contributions, one of the project's priorities is to keep Que as simple, lightweight and dependency-free as possible, and pull requests that change too much or wouldn't be useful to the majority of Que's users have a good chance of being rejected. If you're thinking of submitting a pull request that adds a new feature, consider starting a discussion in [que-talk](https://groups.google.com/forum/#!forum/que-talk) first about what it would do and how it would be implemented. If it's a sufficiently large feature, or if most of Que's users wouldn't find it useful, it may be best implemented as a standalone gem, like some of the related projects above.
|
191
200
|
|
@@ -193,12 +202,54 @@ Regarding contributions, one of the project's priorities is to keep Que as simpl
|
|
193
202
|
|
194
203
|
A note on running specs - Que's worker system is multithreaded and therefore prone to race conditions. As such, if you've touched that code, a single spec run passing isn't a guarantee that any changes you've made haven't introduced bugs. One thing I like to do before pushing changes is rerun the specs many times and watching for hangs. You can do this from the command line with something like:
|
195
204
|
|
196
|
-
|
205
|
+
```bash
|
206
|
+
for i in {1..1000}; do SEED=$i bundle exec rake; done
|
207
|
+
```
|
197
208
|
|
198
209
|
This will iterate the specs one thousand times, each with a different ordering. If the specs hang, note what the seed number was on that iteration. For example, if the previous specs finished with a "Randomized with seed 328", you know that there's a hang with seed 329, and you can narrow it down to a specific spec with:
|
199
210
|
|
200
|
-
|
211
|
+
```bash
|
212
|
+
for i in {1..1000}; do LOG_SPEC=true SEED=328 bundle exec rake; done
|
213
|
+
```
|
201
214
|
|
202
215
|
Note that we iterate because there's no guarantee that the hang would reappear with a single additional run, so we need to rerun the specs until it reappears. The LOG_SPEC parameter will output the name and file location of each spec before it is run, so you can easily tell which spec is hanging, and you can continue narrowing things down from there.
|
203
216
|
|
204
217
|
Another helpful technique is to replace an `it` spec declaration with `hit` - this will run that particular spec 100 times during the run.
|
218
|
+
|
219
|
+
#### With Docker
|
220
|
+
|
221
|
+
We've provided a Dockerised environment to avoid the need to manually: install Ruby, install the gem bundle, set up Postgres, and connect to the database.
|
222
|
+
|
223
|
+
To run the specs using this environment, run:
|
224
|
+
|
225
|
+
```bash
|
226
|
+
./auto/test
|
227
|
+
```
|
228
|
+
|
229
|
+
To get a shell in the environment, run:
|
230
|
+
|
231
|
+
```bash
|
232
|
+
./auto/dev
|
233
|
+
```
|
234
|
+
|
235
|
+
The [Docker Compose config](docker-compose.yml) provides a convenient way to inject your local shell aliases into the Docker container. Simply create a file containing your alias definitions (or which sources them from other files) at `~/.docker-rc.d/.docker-bashrc`, and they will be available inside the container.
|
236
|
+
|
237
|
+
#### Without Docker
|
238
|
+
|
239
|
+
You'll need to have Postgres running. Assuming you have it running on port 5697, with a `que-test` database, and a username & password of `que`, you can run:
|
240
|
+
|
241
|
+
```bash
|
242
|
+
DATABASE_URL=postgres://que:que@localhost:5697/que-test bundle exec rake
|
243
|
+
```
|
244
|
+
|
245
|
+
If you don't already have Postgres, you could use Docker Compose to run just the database:
|
246
|
+
|
247
|
+
```bash
|
248
|
+
docker compose up -d db
|
249
|
+
```
|
250
|
+
|
251
|
+
If you want to try a different version of Postgres, e.g. 12:
|
252
|
+
|
253
|
+
```bash
|
254
|
+
export POSTGRES_VERSION=12
|
255
|
+
```
|
data/auto/dev
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
#!/bin/bash
|
2
|
+
#
|
3
|
+
# Operate in development environment
|
4
|
+
|
5
|
+
set -Eeuo pipefail
|
6
|
+
|
7
|
+
cd "$(dirname "$0")/.."
|
8
|
+
|
9
|
+
docker compose build dev
|
10
|
+
|
11
|
+
# Delete containers and DB volume afterwards on CI
|
12
|
+
if [[ "${CI-}" == "true" ]]; then
|
13
|
+
trap '{
|
14
|
+
echo "Stopping containers..."
|
15
|
+
docker compose down
|
16
|
+
docker volume rm -f que_db-data
|
17
|
+
}' EXIT
|
18
|
+
fi
|
19
|
+
|
20
|
+
set -x
|
21
|
+
docker compose run --rm dev "${@-bash}"
|
data/auto/psql
ADDED
data/auto/test
ADDED
@@ -0,0 +1,17 @@
|
|
1
|
+
#!/bin/bash
|
2
|
+
|
3
|
+
set -Eeuo pipefail
|
4
|
+
|
5
|
+
export POSTGRES_VERSION=14
|
6
|
+
|
7
|
+
delete_db() {
|
8
|
+
docker compose down
|
9
|
+
docker volume rm -f que_db-data
|
10
|
+
}
|
11
|
+
|
12
|
+
trap delete_db EXIT
|
13
|
+
|
14
|
+
# pre-test cleanup is necessary as the existing db container will be used if it's running (potentially with the wrong PG version)
|
15
|
+
delete_db
|
16
|
+
"$(dirname "$0")"/test "$@"
|
17
|
+
delete_db
|
data/docker-compose.yml
ADDED
@@ -0,0 +1,46 @@
|
|
1
|
+
version: "3.7"
|
2
|
+
|
3
|
+
services:
|
4
|
+
|
5
|
+
dev:
|
6
|
+
build:
|
7
|
+
context: .
|
8
|
+
target: dev-environment
|
9
|
+
depends_on:
|
10
|
+
- db
|
11
|
+
volumes:
|
12
|
+
- .:/work
|
13
|
+
- ruby-2.7.5-gem-cache:/usr/local/bundle
|
14
|
+
- ~/.docker-rc.d/:/.docker-rc.d/:ro
|
15
|
+
working_dir: /work
|
16
|
+
entrypoint: /work/scripts/docker-entrypoint
|
17
|
+
command: bash
|
18
|
+
environment:
|
19
|
+
DATABASE_URL: postgres://que:que@db/que-test
|
20
|
+
|
21
|
+
db:
|
22
|
+
image: "postgres:${POSTGRES_VERSION-13}"
|
23
|
+
volumes:
|
24
|
+
- db-data:/var/lib/postgresql/data
|
25
|
+
environment:
|
26
|
+
POSTGRES_USER: que
|
27
|
+
POSTGRES_PASSWORD: que
|
28
|
+
POSTGRES_DB: que-test
|
29
|
+
ports:
|
30
|
+
- 5697:5432
|
31
|
+
|
32
|
+
pg-dev:
|
33
|
+
image: "postgres:${POSTGRES_VERSION-13}"
|
34
|
+
depends_on:
|
35
|
+
- db
|
36
|
+
entrypoint: []
|
37
|
+
command: psql
|
38
|
+
environment:
|
39
|
+
PGHOST: db
|
40
|
+
PGUSER: que
|
41
|
+
PGPASSWORD: que
|
42
|
+
PGDATABASE: que-test
|
43
|
+
|
44
|
+
volumes:
|
45
|
+
db-data: ~
|
46
|
+
ruby-2.7.5-gem-cache: ~
|
data/lib/que/job.rb
CHANGED
@@ -57,22 +57,18 @@ module Que
|
|
57
57
|
|
58
58
|
def enqueue(
|
59
59
|
*args,
|
60
|
-
|
61
|
-
priority: nil,
|
62
|
-
run_at: nil,
|
63
|
-
job_class: nil,
|
64
|
-
tags: nil,
|
60
|
+
job_options: {},
|
65
61
|
**arg_opts
|
66
62
|
)
|
67
|
-
|
63
|
+
arg_opts, job_options = _extract_job_options(arg_opts, job_options.dup)
|
68
64
|
args << arg_opts if arg_opts.any?
|
69
65
|
|
70
|
-
if tags
|
71
|
-
if tags.length > MAXIMUM_TAGS_COUNT
|
72
|
-
raise Que::Error, "Can't enqueue a job with more than #{MAXIMUM_TAGS_COUNT} tags! (passed #{tags.length})"
|
66
|
+
if job_options[:tags]
|
67
|
+
if job_options[:tags].length > MAXIMUM_TAGS_COUNT
|
68
|
+
raise Que::Error, "Can't enqueue a job with more than #{MAXIMUM_TAGS_COUNT} tags! (passed #{job_options[:tags].length})"
|
73
69
|
end
|
74
70
|
|
75
|
-
tags.each do |tag|
|
71
|
+
job_options[:tags].each do |tag|
|
76
72
|
if tag.length > MAXIMUM_TAG_LENGTH
|
77
73
|
raise Que::Error, "Can't enqueue a job with a tag longer than 100 characters! (\"#{tag}\")"
|
78
74
|
end
|
@@ -80,13 +76,13 @@ module Que
|
|
80
76
|
end
|
81
77
|
|
82
78
|
attrs = {
|
83
|
-
queue: queue || resolve_que_setting(:queue) || Que.default_queue,
|
84
|
-
priority: priority || resolve_que_setting(:priority),
|
85
|
-
run_at: run_at || resolve_que_setting(:run_at),
|
79
|
+
queue: job_options[:queue] || resolve_que_setting(:queue) || Que.default_queue,
|
80
|
+
priority: job_options[:priority] || resolve_que_setting(:priority),
|
81
|
+
run_at: job_options[:run_at] || resolve_que_setting(:run_at),
|
86
82
|
args: Que.serialize_json(args),
|
87
|
-
data: tags ? Que.serialize_json(tags: tags) : "{}",
|
83
|
+
data: job_options[:tags] ? Que.serialize_json(tags: job_options[:tags]) : "{}",
|
88
84
|
job_class: \
|
89
|
-
job_class || name ||
|
85
|
+
job_options[:job_class] || name ||
|
90
86
|
raise(Error, "Can't enqueue an anonymous subclass of Que::Job"),
|
91
87
|
}
|
92
88
|
|
@@ -139,6 +135,27 @@ module Que
|
|
139
135
|
end
|
140
136
|
end
|
141
137
|
end
|
138
|
+
|
139
|
+
def _extract_job_options(arg_opts, job_options)
|
140
|
+
deprecated_job_option_names = []
|
141
|
+
|
142
|
+
%i[queue priority run_at job_class tags].each do |option_name|
|
143
|
+
next unless arg_opts.key?(option_name) && job_options[option_name].nil?
|
144
|
+
|
145
|
+
job_options[option_name] = arg_opts.delete(option_name)
|
146
|
+
deprecated_job_option_names << option_name
|
147
|
+
end
|
148
|
+
|
149
|
+
_log_job_options_deprecation(deprecated_job_option_names)
|
150
|
+
|
151
|
+
[arg_opts, job_options]
|
152
|
+
end
|
153
|
+
|
154
|
+
def _log_job_options_deprecation(deprecated_job_option_names)
|
155
|
+
return unless deprecated_job_option_names.any?
|
156
|
+
|
157
|
+
warn "Passing job options like (#{deprecated_job_option_names.join(', ')}) to `JobClass.enqueue` as top level keyword args has been deprecated and will be removed in version 2.0. Please wrap job options in an explicit `job_options` keyword arg instead."
|
158
|
+
end
|
142
159
|
end
|
143
160
|
|
144
161
|
# Set up some defaults.
|
data/lib/que/version.rb
CHANGED
data/lib/que/worker.rb
CHANGED
@@ -137,6 +137,7 @@ module Que
|
|
137
137
|
error: {
|
138
138
|
class: error.class.to_s,
|
139
139
|
message: error.message,
|
140
|
+
backtrace: (error.backtrace || []).join("\n").slice(0, 10000),
|
140
141
|
},
|
141
142
|
)
|
142
143
|
|
@@ -164,7 +165,7 @@ module Que
|
|
164
165
|
Que.execute :set_error, [
|
165
166
|
delay,
|
166
167
|
"#{error.class}: #{error.message}".slice(0, 500),
|
167
|
-
error.backtrace.join("\n").slice(0, 10000),
|
168
|
+
(error.backtrace || []).join("\n").slice(0, 10000),
|
168
169
|
job.fetch(:id),
|
169
170
|
]
|
170
171
|
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
#!/bin/bash
|
2
|
+
|
3
|
+
set -Eeuo pipefail
|
4
|
+
|
5
|
+
# For using your own dotfiles within the Docker container
|
6
|
+
if [ -f /.docker-rc.d/.docker-bashrc ]; then
|
7
|
+
echo "source /.docker-rc.d/.docker-bashrc" >> ~/.bashrc
|
8
|
+
fi
|
9
|
+
|
10
|
+
gem list -i -e bundler -v "$RUBY_BUNDLER_VERSION" >/dev/null || gem install bundler -v "$RUBY_BUNDLER_VERSION"
|
11
|
+
|
12
|
+
bundle check --dry-run || bundle install
|
13
|
+
|
14
|
+
exec "${@-bash}"
|
data/scripts/test
ADDED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: que
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Chris Hanks
|
8
|
-
autorequire:
|
8
|
+
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2022-02-23 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -34,13 +34,18 @@ extra_rdoc_files: []
|
|
34
34
|
files:
|
35
35
|
- ".github/workflows/tests.yml"
|
36
36
|
- ".gitignore"
|
37
|
-
- CHANGELOG.1.0.beta.md
|
38
37
|
- CHANGELOG.md
|
38
|
+
- Dockerfile
|
39
39
|
- LICENSE.txt
|
40
40
|
- README.md
|
41
41
|
- Rakefile
|
42
|
+
- auto/dev
|
43
|
+
- auto/psql
|
44
|
+
- auto/test
|
45
|
+
- auto/test-postgres-14
|
42
46
|
- bin/command_line_interface.rb
|
43
47
|
- bin/que
|
48
|
+
- docker-compose.yml
|
44
49
|
- docs/README.md
|
45
50
|
- lib/que.rb
|
46
51
|
- lib/que/active_job/extensions.rb
|
@@ -80,11 +85,13 @@ files:
|
|
80
85
|
- lib/que/version.rb
|
81
86
|
- lib/que/worker.rb
|
82
87
|
- que.gemspec
|
88
|
+
- scripts/docker-entrypoint
|
89
|
+
- scripts/test
|
83
90
|
homepage: https://github.com/que-rb/que
|
84
91
|
licenses:
|
85
92
|
- MIT
|
86
93
|
metadata: {}
|
87
|
-
post_install_message:
|
94
|
+
post_install_message:
|
88
95
|
rdoc_options: []
|
89
96
|
require_paths:
|
90
97
|
- lib
|
@@ -95,12 +102,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
95
102
|
version: '0'
|
96
103
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
97
104
|
requirements:
|
98
|
-
- - "
|
105
|
+
- - ">="
|
99
106
|
- !ruby/object:Gem::Version
|
100
|
-
version:
|
107
|
+
version: '0'
|
101
108
|
requirements: []
|
102
109
|
rubygems_version: 3.1.6
|
103
|
-
signing_key:
|
110
|
+
signing_key:
|
104
111
|
specification_version: 4
|
105
112
|
summary: A PostgreSQL-based Job Queue
|
106
113
|
test_files: []
|
data/CHANGELOG.1.0.beta.md
DELETED
@@ -1,137 +0,0 @@
|
|
1
|
-
### 1.0.0.beta3 (2018-05-18)
|
2
|
-
|
3
|
-
* Added support for customizing log levels for `job_worked` events (#217).
|
4
|
-
|
5
|
-
* Began logging all `job_errored` events at the `ERROR` log level.
|
6
|
-
|
7
|
-
* Fixed the Railtie when running in test mode (#214).
|
8
|
-
|
9
|
-
* Tweaked the meanings of worker-priorities and worker-count options in the CLI, to better support use cases with low worker counts (#216).
|
10
|
-
|
11
|
-
### 1.0.0.beta2 (2018-04-13)
|
12
|
-
|
13
|
-
* Fixed an incompatibility that caused the new locker to hang when using Rails in development mode (#213).
|
14
|
-
|
15
|
-
* Fixed a bug with setting the log level via the CLI when the configured logger was based on a callable (#210).
|
16
|
-
|
17
|
-
* Renamed Que.middleware to Que.job_middleware.
|
18
|
-
|
19
|
-
* Added Que.sql_middleware.
|
20
|
-
|
21
|
-
* Officially added support for Ruby 2.5.
|
22
|
-
|
23
|
-
* Internal cleanup and renamings.
|
24
|
-
|
25
|
-
### 1.0.0.beta (2017-10-25)
|
26
|
-
|
27
|
-
* **A schema upgrade to version 4 will be required for this release.** See [the migration doc](https://github.com/que-rb/que/blob/master/docs/migrating.md) for information if you're upgrading from a previous release.
|
28
|
-
|
29
|
-
* Please note that this migration requires a rewrite of the jobs table, which makes it O(n) with the size of the table. If you have a very large backlog of jobs you may want to schedule downtime for this migration.
|
30
|
-
|
31
|
-
* Que's implementation has been changed from one in which worker threads hold their own PG connections and lock their own jobs to one in which a single thread (and PG connection) locks jobs through LISTEN/NOTIFY and batch polling, and passes jobs along to worker threads. This has many benefits, including:
|
32
|
-
|
33
|
-
* Jobs queued for immediate processing can be actively distributed to workers with LISTEN/NOTIFY, which is more efficient than having workers repeatedly poll for new jobs.
|
34
|
-
|
35
|
-
* When polling is necessary (to pick up jobs that are scheduled for the future or that need to be retried due to errors), jobs can be locked and fetched in batches, rather than one at a time.
|
36
|
-
|
37
|
-
* Individual workers no longer need to monopolize their own (usually idle) connections while working jobs, so Ruby processes will require fewer Postgres connections.
|
38
|
-
|
39
|
-
* PgBouncer or another external connection pool can be used for workers' connections (though not for the connection used to lock and listen for jobs).
|
40
|
-
|
41
|
-
* Other features introduced in this version include:
|
42
|
-
|
43
|
-
* Much better support for all versions of ActiveJob.
|
44
|
-
|
45
|
-
* In particular, you may (optionally) include `Que::ActiveJob::JobExtensions` into `ApplicationJob` to get support for all of Que's job helper methods.
|
46
|
-
|
47
|
-
* Custom middleware that wrap running jobs are now supported.
|
48
|
-
|
49
|
-
* Support for categorizing jobs with tags.
|
50
|
-
|
51
|
-
* Support for configuring a `maximum_retry_count` on individual job classes.
|
52
|
-
|
53
|
-
* Job configuration options are now inheritable, so job class hierarchies are more useful.
|
54
|
-
|
55
|
-
* There are now built-in models for ActiveRecord and Sequel to allow inspecting the queue easily.
|
56
|
-
|
57
|
-
* Jobs that have finished working may optionally be retained in the database indefinitely.
|
58
|
-
|
59
|
-
* To keep a job record, replace the `destroy` calls in your jobs with `finish`. `destroy` will still delete records entirely, for jobs that you don't want to keep.
|
60
|
-
|
61
|
-
* If you don't resolve a job yourself one way or another, Que will still `destroy` the job for you by default.
|
62
|
-
|
63
|
-
* Finished jobs have a timestamp set in the finished_at column.
|
64
|
-
|
65
|
-
* Jobs that have errored too many times will now be marked as expired, and won't be retried again.
|
66
|
-
|
67
|
-
* You can configure a maximum_retry_count in your job classes, to set the threshold at which a job will be marked expired. The default is 15.
|
68
|
-
|
69
|
-
* To manually mark a job as expired (and keep it in the database but not try to run it again) you can call `expire` helper in your job.
|
70
|
-
|
71
|
-
* You can now set job priority thresholds for individual workers, to ensure that there will always be space available for high-priority jobs.
|
72
|
-
|
73
|
-
* `Que.job_states` returns a list of locked jobs and the hostname/pid of the Ruby processes that have locked them.
|
74
|
-
|
75
|
-
* `Que.connection_proc=` has been added, to allow for the easy integration of custom connection pools.
|
76
|
-
|
77
|
-
* In keeping with semantic versioning, the major version is being bumped since the new implementation requires some backwards-incompatible changes. These changes include:
|
78
|
-
|
79
|
-
* Support for MRI Rubies before 2.2 has been dropped.
|
80
|
-
|
81
|
-
* Support for Postgres versions before 9.5 has been dropped (JSONB and upsert support is required).
|
82
|
-
|
83
|
-
* JRuby support has been dropped. It will be reintroduced whenever the jruby-pg gem is production-ready.
|
84
|
-
|
85
|
-
* The `que:work` rake task has been removed. Use the `que` executable instead.
|
86
|
-
|
87
|
-
* Therefore, configuring workers using QUE_* environment variables is no longer supported. Please pass the appropriate options to the `que` executable instead.
|
88
|
-
|
89
|
-
* The `mode` setter has been removed.
|
90
|
-
|
91
|
-
* To run jobs synchronously when they are enqueued (the old `:sync` behavior) you can set `Que.run_synchronously = true`.
|
92
|
-
|
93
|
-
* To start up the worker pool (the old :async behavior) you should use the `que` executable to start up a worker process. There's no longer a supported API for running workers outside of the `que` executable.
|
94
|
-
|
95
|
-
* The following methods are not meaningful under the new implementation and have been removed:
|
96
|
-
|
97
|
-
* The `Que.wake_interval` getter and setter.
|
98
|
-
|
99
|
-
* The `Que.worker_count` getter and setter.
|
100
|
-
|
101
|
-
* `Que.wake!`
|
102
|
-
|
103
|
-
* `Que.wake_all!`
|
104
|
-
|
105
|
-
* Since Que needs a dedicated Postgres connection to manage job locks, running Que through a single PG connection is no longer supported.
|
106
|
-
|
107
|
-
* It's not clear that anyone ever actually did this.
|
108
|
-
|
109
|
-
* `Que.worker_states` has been removed, as the connection that locks a job is no longer the one that the job is using to run. Its functionality has been partially replaced with `Que.job_states`.
|
110
|
-
|
111
|
-
* When using Rails, for simplicity, job attributes and keys in argument hashes are now converted to symbols when retrieved from the database, rather than being converted to instances of HashWithIndifferentAccess.
|
112
|
-
|
113
|
-
* Arguments passed to jobs are now deep-frozen, to prevent unexpected behavior when the args are mutated and the job is reenqueued.
|
114
|
-
|
115
|
-
* Since JSONB is now used to store arguments, the order of argument hashes is no longer maintained.
|
116
|
-
|
117
|
-
* It wouldn't have been a good idea to rely on this anyway.
|
118
|
-
|
119
|
-
* Calling Que.log() directly is no longer supported/recommended.
|
120
|
-
|
121
|
-
* Features marked as deprecated in the final 0.x releases have been removed.
|
122
|
-
|
123
|
-
* Finally, if you've built up your own tooling and customizations around Que, you may need to be aware of some DB schema changes made in the migration to schema version #4.
|
124
|
-
|
125
|
-
* The `job_id` column has been renamed `id` and is now the primary key. This makes it easier to manage the queue using an ActiveRecord model.
|
126
|
-
|
127
|
-
* Finished jobs are now kept in the DB, unless you explicitly call `destroy`. If you want to query the DB for only jobs that haven't finished yet, add a `WHERE finished_at IS NULL` condition to your query, or use the not_finished scope on one of the provided ORM models.
|
128
|
-
|
129
|
-
* There is now an `expired_at` timestamp column, which is set when a job reaches its maximum number of retries and will not be attempted again.
|
130
|
-
|
131
|
-
* Due to popular demand, the default queue name is now "default" rather than an empty string. The migration will move pending jobs under the "" queue to the "default" queue.
|
132
|
-
|
133
|
-
* The `last_error` column has been split in two, to `last_error_message` and `last_error_backtrace`. These two columns are now limited to 500 and 10,000 characters, respectively. The migration will split old error data correctly, and truncate it if necessary.
|
134
|
-
|
135
|
-
* Names for queues and job classes are now limited to 500 characters, which is still far longer than either of these values should reasonably be.
|
136
|
-
|
137
|
-
* There is now a `data` JSONB column which is used to support various ways of organizing jobs (setting tags on them, etc).
|