queue_classic 3.2.0.RC1 → 4.0.0.pre.alpha1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.circleci/config.yml +151 -0
- data/.gitignore +2 -0
- data/{changelog → CHANGELOG.md} +80 -34
- data/CODE_OF_CONDUCT.md +46 -0
- data/Gemfile +8 -5
- data/README.md +77 -85
- data/Rakefile +2 -0
- data/lib/generators/queue_classic/install_generator.rb +6 -0
- data/lib/generators/queue_classic/templates/add_queue_classic.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_0_0.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_0_2.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_1_0.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_4_0_0.rb +11 -0
- data/lib/queue_classic.rb +4 -11
- data/lib/queue_classic/config.rb +2 -1
- data/lib/queue_classic/conn_adapter.rb +28 -14
- data/lib/queue_classic/queue.rb +65 -11
- data/lib/queue_classic/railtie.rb +2 -0
- data/lib/queue_classic/setup.rb +24 -7
- data/lib/queue_classic/tasks.rb +4 -5
- data/lib/queue_classic/version.rb +3 -1
- data/lib/queue_classic/worker.rb +10 -5
- data/queue_classic.gemspec +1 -1
- data/sql/create_table.sql +7 -16
- data/sql/ddl.sql +6 -82
- data/sql/downgrade_from_4_0_0.sql +88 -0
- data/sql/update_to_3_0_0.sql +5 -5
- data/sql/update_to_3_1_0.sql +6 -6
- data/sql/update_to_4_0_0.sql +6 -0
- data/test/benchmark_test.rb +2 -0
- data/test/config_test.rb +2 -0
- data/test/helper.rb +34 -0
- data/test/lib/queue_classic_rails_connection_test.rb +9 -6
- data/test/lib/queue_classic_test.rb +2 -0
- data/test/queue_test.rb +62 -2
- data/test/rails-tests/.gitignore +2 -0
- data/test/rails-tests/rails523.sh +23 -0
- data/test/worker_test.rb +138 -17
- metadata +15 -7
- data/.travis.yml +0 -15
data/README.md
CHANGED
@@ -1,41 +1,47 @@
|
|
1
1
|
# queue_classic
|
2
|
+
A simple, efficient worker queue for Ruby & PostgreSQL
|
2
3
|
|
3
|
-
|
4
|
-
<b>Simple, efficient worker queue for Ruby & PostgreSQL</b>
|
5
|
-
<br />
|
6
|
-
<a href="https://travis-ci.org/QueueClassic/queue_classic"><img src="http://img.shields.io/travis/QueueClassic/queue_classic/master.svg?style=flat" /></a>
|
4
|
+
[![CircleCI](https://circleci.com/gh/QueueClassic/queue_classic/tree/master.svg?style=svg)](https://circleci.com/gh/QueueClassic/queue_classic/tree/master)
|
7
5
|
|
8
|
-
|
6
|
+
[![Gem Version](http://img.shields.io/gem/v/queue_classic.svg?style=flat)](http://badge.fury.io/rb/queue_classic)
|
9
7
|
|
10
|
-
|
11
|
-
</p>
|
8
|
+
**IMPORTANT NOTE: This README is representing the current work for queue_classic, which is generally the pending next version. You can find the README for other versions:**
|
12
9
|
|
10
|
+
You can always find the latest and previous releases here:
|
13
11
|
|
14
|
-
|
12
|
+
https://github.com/QueueClassic/queue_classic/releases
|
15
13
|
|
16
|
-
|
14
|
+
## What is queue_classic?
|
17
15
|
|
18
|
-
-
|
19
|
-
- latest stable can be found: [v3.0.X](https://github.com/QueueClassic/queue_classic/tree/3-0-stable)
|
20
|
-
- older stable: [v2.2.3](https://github.com/QueueClassic/queue_classic/tree/v2.2.3)
|
16
|
+
queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.
|
21
17
|
|
18
|
+
A major benefit is the ability to enqueue inside transactions, ensuring things are done only when your changes are commited.
|
22
19
|
|
23
|
-
##
|
20
|
+
## Other related projects
|
24
21
|
|
25
|
-
|
22
|
+
* [Queue Classic Plus](https://github.com/rainforestapp/queue_classic_plus) - adds support for retrying with specific exceptions, transaction processing of jobs, metric collection, etc
|
23
|
+
* [Queue Classic Admin](https://github.com/QueueClassic/queue_classic_admin) - Admin interface for queue_classic
|
24
|
+
* [Queue Classic Matchers](https://github.com/rainforestapp/queue_classic_matchers) - RSpec matchers for queue_classic
|
26
25
|
|
27
26
|
## Features
|
28
27
|
|
29
|
-
* Leverage of PostgreSQL's listen/notify
|
28
|
+
* Leverage of PostgreSQL's listen/notify, skip locked, and row locking.
|
30
29
|
* Support for multiple queues with heterogeneous workers.
|
31
30
|
* JSON data format.
|
32
|
-
* Forking workers.
|
33
31
|
* Workers can work multiple queues.
|
34
|
-
*
|
32
|
+
* ~~Forking workers.~~ (currently apparently broken https://github.com/QueueClassic/queue_classic/issues/207, a WIP fix exists but isn't merged tested yet https://github.com/QueueClassic/queue_classic/pull/216)
|
33
|
+
|
34
|
+
### Requirements
|
35
|
+
|
36
|
+
For this version, the requirements are as follows:
|
37
|
+
* Ruby 2.4, 2.5 or 2.6
|
38
|
+
* Postgres ~> 9.6
|
39
|
+
* Rubygem: pg ~> 0.17
|
35
40
|
|
36
|
-
## Table of content
|
37
41
|
|
38
|
-
|
42
|
+
## Table of contents
|
43
|
+
|
44
|
+
* [Documentation](https://www.rubydoc.info/gems/queue_classic/)
|
39
45
|
* [Usage](#usage)
|
40
46
|
* [Setup](#setup)
|
41
47
|
* [Upgrade from earlier versions to V3.1](#upgrade-from-earlier-versions)
|
@@ -48,14 +54,14 @@ queue_classic provides a simple interface to a PostgreSQL-backed message queue.
|
|
48
54
|
|
49
55
|
## Usage
|
50
56
|
|
51
|
-
There are 2 ways to use queue_classic
|
57
|
+
There are 2 ways to use queue_classic:
|
52
58
|
|
53
59
|
* Producing Jobs
|
54
60
|
* Working Jobs
|
55
61
|
|
56
62
|
### Producing Jobs
|
57
63
|
|
58
|
-
The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method
|
64
|
+
The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method defined by the first argument. The set of arguments will be encoded as JSON and stored in the database.
|
59
65
|
|
60
66
|
```ruby
|
61
67
|
# This method has no arguments.
|
@@ -78,13 +84,13 @@ p_queue = QC::Queue.new("priority_queue")
|
|
78
84
|
p_queue.enqueue("Kernel.puts", ["hello", "world"])
|
79
85
|
```
|
80
86
|
|
81
|
-
There is also the
|
87
|
+
There is also the ability to schedule a job to run at a specified time in the future. The job will become processable after the specified time, and will be processed as-soon-as-possible.
|
82
88
|
|
83
89
|
```ruby
|
84
|
-
#
|
90
|
+
# Specify the job execution time exactly
|
85
91
|
QC.enqueue_at(Time.new(2024,01,02,10,00), "Kernel.puts", "hello future")
|
86
92
|
|
87
|
-
#
|
93
|
+
# Specify the job execution time as an offset in seconds
|
88
94
|
QC.enqueue_in(60, "Kernel.puts", "hello from 1 minute later")
|
89
95
|
```
|
90
96
|
|
@@ -94,28 +100,33 @@ There are two ways to work jobs. The first approach is to use the Rake task. The
|
|
94
100
|
|
95
101
|
#### Rake Task
|
96
102
|
|
97
|
-
Require queue_classic in your Rakefile
|
103
|
+
Require queue_classic in your Rakefile:
|
98
104
|
|
99
105
|
```ruby
|
100
106
|
require 'queue_classic'
|
101
107
|
require 'queue_classic/tasks'
|
102
108
|
```
|
103
109
|
|
104
|
-
|
110
|
+
##### Work all queues
|
111
|
+
Start the worker via the Rakefile:
|
112
|
+
|
105
113
|
```bash
|
106
114
|
$ bundle exec rake qc:work
|
107
115
|
```
|
108
116
|
|
109
|
-
|
117
|
+
##### Work a single specific queue
|
118
|
+
Setup a worker to work only a specific, non-default queue:
|
119
|
+
|
110
120
|
```bash
|
111
121
|
$ QUEUE="priority_queue" bundle exec rake qc:work
|
112
122
|
```
|
113
123
|
|
114
|
-
|
124
|
+
##### Work multiple queues
|
125
|
+
In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
|
126
|
+
|
115
127
|
```bash
|
116
128
|
$ QUEUES="priority_queue,secondary_queue" bundle exec rake qc:work
|
117
129
|
```
|
118
|
-
In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
|
119
130
|
|
120
131
|
#### Custom Worker
|
121
132
|
|
@@ -128,7 +139,6 @@ require 'queue_classic'
|
|
128
139
|
FailedQueue = QC::Queue.new("failed_jobs")
|
129
140
|
|
130
141
|
class MyWorker < QC::Worker
|
131
|
-
|
132
142
|
# A job is a Hash containing these attributes:
|
133
143
|
# :id Integer, the job id
|
134
144
|
# :method String, containing the object and method
|
@@ -141,7 +151,7 @@ class MyWorker < QC::Worker
|
|
141
151
|
# Do something with the job
|
142
152
|
...
|
143
153
|
end
|
144
|
-
|
154
|
+
|
145
155
|
# This method will be called when an exception
|
146
156
|
# is raised during the execution of the job.
|
147
157
|
# First argument is the job that failed.
|
@@ -149,7 +159,6 @@ class MyWorker < QC::Worker
|
|
149
159
|
def handle_failure(job, e)
|
150
160
|
FailedQueue.enqueue(job[:method], *job[:args])
|
151
161
|
end
|
152
|
-
|
153
162
|
end
|
154
163
|
|
155
164
|
worker = MyWorker.new
|
@@ -158,13 +167,12 @@ trap('INT') { exit }
|
|
158
167
|
trap('TERM') { worker.stop }
|
159
168
|
|
160
169
|
loop do
|
161
|
-
job = worker.lock_job
|
162
|
-
Timeout::timeout(5) { worker.process(job) }
|
170
|
+
queue, job = worker.lock_job
|
171
|
+
Timeout::timeout(5) { worker.process(queue, job) }
|
163
172
|
end
|
164
173
|
```
|
165
174
|
|
166
|
-
The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to
|
167
|
-
inject your own worker class:
|
175
|
+
The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to inject your own worker class:
|
168
176
|
|
169
177
|
```ruby
|
170
178
|
QC.default_worker_class = MyWorker
|
@@ -172,7 +180,7 @@ QC.default_worker_class = MyWorker
|
|
172
180
|
|
173
181
|
## Setup
|
174
182
|
|
175
|
-
In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `
|
183
|
+
In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `psql` or use a database migration script.
|
176
184
|
|
177
185
|
### Quick Start
|
178
186
|
|
@@ -187,28 +195,37 @@ $ ruby -r queue_classic -e "QC::Worker.new.work"
|
|
187
195
|
|
188
196
|
### Ruby on Rails Setup
|
189
197
|
|
190
|
-
Declare dependencies in Gemfile
|
198
|
+
Declare dependencies in Gemfile:
|
199
|
+
|
191
200
|
```ruby
|
192
|
-
source
|
193
|
-
gem
|
201
|
+
source 'https://rubygems.org' do
|
202
|
+
gem 'queue_classic'
|
203
|
+
end
|
194
204
|
```
|
195
205
|
|
196
|
-
|
206
|
+
Install queue_classic, which adds the needed migrations for the database tables and stored procedures:
|
197
207
|
|
198
|
-
```
|
208
|
+
```bash
|
199
209
|
rails generate queue_classic:install
|
200
210
|
bundle exec rake db:migrate
|
201
211
|
```
|
202
212
|
|
213
|
+
#### Database connection
|
214
|
+
|
215
|
+
Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used. If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
|
216
|
+
|
217
|
+
**Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
|
218
|
+
|
203
219
|
#### Active Job
|
204
220
|
|
205
|
-
If you use Rails 4.2
|
221
|
+
If you use Rails 4.2+ and want to use Active Job, all you need to do is to set `config.active_job.queue_adapter = :queue_classic` in your `application.rb`. Everything else will be taken care for you. You can now use the Active Job functionality from now.
|
206
222
|
|
207
223
|
Just for your information, queue_classic detects your database connection and uses it.
|
208
224
|
|
209
|
-
### Rake Task Setup
|
210
225
|
|
211
|
-
|
226
|
+
### Plain Ruby Setup
|
227
|
+
|
228
|
+
If you're not using rails, you can use the Rake task to prepare your database:
|
212
229
|
|
213
230
|
```bash
|
214
231
|
# Creating the table and functions
|
@@ -218,35 +235,26 @@ $ bundle exec rake qc:create
|
|
218
235
|
$ bundle exec rake qc:drop
|
219
236
|
```
|
220
237
|
|
221
|
-
|
238
|
+
#### Database connection
|
222
239
|
|
223
|
-
|
240
|
+
By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
|
224
241
|
|
225
|
-
|
242
|
+
## Upgrading from earlier versions
|
226
243
|
|
227
|
-
If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
|
228
|
-
|
229
|
-
**Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
|
230
|
-
|
231
|
-
|
232
|
-
#### Other Ruby apps
|
233
|
-
|
234
|
-
By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
|
235
|
-
|
236
|
-
## Upgrade from earlier versions
|
237
244
|
If you are upgrading from a previous version of queue_classic, you might need some new database columns and/or functions. Luckily enough for you, it is easy to do so.
|
238
245
|
|
239
246
|
### Ruby on Rails
|
240
247
|
|
241
|
-
|
248
|
+
These two commands will add the newer migrations:
|
242
249
|
|
243
|
-
```
|
250
|
+
```bash
|
244
251
|
rails generate queue_classic:install
|
245
252
|
bundle exec rake db:migrate
|
246
253
|
```
|
254
|
+
|
247
255
|
### Rake Task
|
248
256
|
|
249
|
-
This rake task will
|
257
|
+
This rake task will update you to the latest version:
|
250
258
|
```bash
|
251
259
|
# Updating the table and functions
|
252
260
|
$ bundle exec rake qc:update
|
@@ -256,30 +264,24 @@ $ bundle exec rake qc:update
|
|
256
264
|
|
257
265
|
All configuration takes place in the form of environment vars. See [queue_classic.rb](https://github.com/QueueClassic/queue_classic/blob/master/lib/queue_classic.rb#L23-62) for a list of options.
|
258
266
|
|
259
|
-
## JSON
|
260
|
-
|
261
|
-
If you are running PostgreSQL 9.4 or higher, queue_classic will use the [jsonb](http://www.postgresql.org/docs/9.4/static/datatype-json.html) datatype for new tables. Versions 9.2 and 9.3 will use the `json` data type and versions 9.1 and lower will use the `text` data type.
|
262
|
-
If you are updating queue_classic and are running PostgreSQL >= 9.4, run the following to switch to `jsonb`:
|
263
|
-
```
|
264
|
-
alter table queue_classic_jobs alter column args type jsonb using (args::jsonb);
|
265
|
-
```
|
266
|
-
|
267
267
|
## Logging
|
268
268
|
|
269
269
|
By default queue_classic does not talk very much.
|
270
270
|
If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: DEBUG and MEASURE.
|
271
271
|
|
272
272
|
### Measure
|
273
|
-
This will output the time to process and that kind of thing. To enable it, set the `QC_MEASURE`:
|
274
273
|
|
275
|
-
|
274
|
+
This will output the time to process and some more statistics. To enable it, set the `QC_MEASURE`:
|
275
|
+
|
276
|
+
```bash
|
276
277
|
export QC_MEASURE="true"
|
277
278
|
```
|
278
279
|
|
279
280
|
### Debug
|
281
|
+
|
280
282
|
You can enable the debug output by setting the `DEBUG` environment variable:
|
281
283
|
|
282
|
-
```
|
284
|
+
```bash
|
283
285
|
export DEBUG="true"
|
284
286
|
```
|
285
287
|
|
@@ -287,31 +289,21 @@ export DEBUG="true"
|
|
287
289
|
|
288
290
|
If you think you have found a bug, feel free to open an issue. Use the following template for the new issue:
|
289
291
|
|
290
|
-
1. List
|
292
|
+
1. List your versions: Ruby, PostgreSQL, queue_classic.
|
291
293
|
2. Define what you would have expected to happen.
|
292
294
|
3. List what actually happened.
|
293
295
|
4. Provide sample codes & commands which will reproduce the problem.
|
294
296
|
|
295
|
-
If you have general questions about how to use queue_classic, send a message to the mailing list:
|
296
|
-
|
297
|
-
https://groups.google.com/d/forum/queue_classic
|
298
|
-
|
299
297
|
## Hacking on queue_classic
|
300
298
|
|
301
|
-
### Dependencies
|
302
|
-
|
303
|
-
* Ruby 1.9.2
|
304
|
-
* Postgres ~> 9.0
|
305
|
-
* Rubygem: pg ~> 0.11.0
|
306
|
-
* For JRuby, see [queue_classic_java](https://github.com/bdon/queue_classic_java)
|
307
|
-
|
308
299
|
### Running Tests
|
309
300
|
|
310
301
|
```bash
|
311
302
|
$ bundle
|
312
303
|
$ createdb queue_classic_test
|
313
304
|
$ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
|
314
|
-
$ rake
|
305
|
+
$ bundle exec rake # run all tests
|
306
|
+
$ bundle exec ruby test/queue_test.rb # run a single test
|
315
307
|
```
|
316
308
|
|
317
309
|
## License
|
data/Rakefile
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require 'rails/generators'
|
2
4
|
require 'rails/generators/migration'
|
3
5
|
require 'active_record'
|
@@ -31,6 +33,10 @@ module QC
|
|
31
33
|
if self.class.migration_exists?('db/migrate', 'update_queue_classic_3_1_0').nil?
|
32
34
|
migration_template 'update_queue_classic_3_1_0.rb', 'db/migrate/update_queue_classic_3_1_0.rb'
|
33
35
|
end
|
36
|
+
|
37
|
+
if self.class.migration_exists?('db/migrate', 'update_queue_classic_4_0_0').nil?
|
38
|
+
migration_template 'update_queue_classic_4_0_0.rb', 'db/migrate/update_queue_classic_4_0_0.rb'
|
39
|
+
end
|
34
40
|
end
|
35
41
|
end
|
36
42
|
end
|
data/lib/queue_classic.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require_relative "queue_classic/config"
|
2
4
|
|
3
5
|
module QC
|
@@ -49,15 +51,7 @@ Please use the method QC.#{config_method} instead.
|
|
49
51
|
end
|
50
52
|
|
51
53
|
def self.default_conn_adapter
|
52
|
-
|
53
|
-
return t[:qc_conn_adapter] if t[:qc_conn_adapter]
|
54
|
-
adapter = if rails_connection_sharing_enabled?
|
55
|
-
ConnAdapter.new(ActiveRecord::Base.connection.raw_connection)
|
56
|
-
else
|
57
|
-
ConnAdapter.new
|
58
|
-
end
|
59
|
-
|
60
|
-
t[:qc_conn_adapter] = adapter
|
54
|
+
Thread.current[:qc_conn_adapter] ||= ConnAdapter.new(active_record_connection_share: rails_connection_sharing_enabled?)
|
61
55
|
end
|
62
56
|
|
63
57
|
def self.default_conn_adapter=(conn)
|
@@ -100,8 +94,7 @@ Please use the method QC.#{config_method} instead.
|
|
100
94
|
# This will unlock all jobs any postgres' PID that is not existing anymore
|
101
95
|
# to prevent any infinitely locked jobs
|
102
96
|
def self.unlock_jobs_of_dead_workers
|
103
|
-
|
104
|
-
default_conn_adapter.execute("UPDATE #{QC.table_name} SET locked_at = NULL, locked_by = NULL WHERE locked_by NOT IN (SELECT #{pid_column} FROM pg_stat_activity);")
|
97
|
+
default_conn_adapter.execute("UPDATE #{QC.table_name} SET locked_at = NULL, locked_by = NULL WHERE locked_by NOT IN (SELECT pid FROM pg_stat_activity);")
|
105
98
|
end
|
106
99
|
|
107
100
|
# private class methods
|
data/lib/queue_classic/config.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
module QC
|
2
4
|
module Config
|
3
5
|
# You can use the APP_NAME to query for
|
@@ -58,7 +60,6 @@ module QC
|
|
58
60
|
|
59
61
|
# The worker class instantiated by QC's rake tasks.
|
60
62
|
def default_worker_class
|
61
|
-
|
62
63
|
@worker_class ||= (ENV["QC_DEFAULT_WORKER_CLASS"] && Kernel.const_get(ENV["QC_DEFAULT_WORKER_CLASS"]) ||
|
63
64
|
QC::Worker)
|
64
65
|
|