queue_classic 3.1.0.RC1 → 4.0.0.pre.beta1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +5 -5
- data/.circleci/config.yml +192 -0
- data/.gitignore +11 -0
- data/CHANGELOG.md +192 -0
- data/CODE_OF_CONDUCT.md +46 -0
- data/CONTRIBUTING.md +17 -0
- data/Gemfile +12 -0
- data/LICENSE.txt +20 -0
- data/{readme.md → README.md} +120 -83
- data/Rakefile +16 -0
- data/lib/generators/queue_classic/install_generator.rb +6 -0
- data/lib/generators/queue_classic/templates/add_queue_classic.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_0_0.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_0_2.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_3_1_0.rb +3 -1
- data/lib/generators/queue_classic/templates/update_queue_classic_4_0_0.rb +11 -0
- data/lib/queue_classic/config.rb +86 -0
- data/lib/queue_classic/conn_adapter.rb +37 -16
- data/lib/queue_classic/queue.rb +76 -18
- data/lib/queue_classic/railtie.rb +2 -0
- data/lib/queue_classic/setup.rb +24 -7
- data/lib/queue_classic/tasks.rb +7 -8
- data/lib/queue_classic/version.rb +5 -0
- data/lib/queue_classic/worker.rb +18 -12
- data/lib/queue_classic.rb +50 -58
- data/queue_classic.gemspec +25 -0
- data/sql/create_table.sql +7 -14
- data/sql/ddl.sql +6 -82
- data/sql/downgrade_from_4_0_0.sql +88 -0
- data/sql/update_to_3_0_0.sql +5 -5
- data/sql/update_to_3_1_0.sql +6 -6
- data/sql/update_to_4_0_0.sql +6 -0
- data/test/benchmark_test.rb +15 -12
- data/test/config_test.rb +123 -0
- data/test/helper.rb +47 -3
- data/test/helper.sql +25 -0
- data/test/lib/queue_classic_rails_connection_test.rb +16 -10
- data/test/lib/queue_classic_test.rb +15 -3
- data/test/lib/queue_classic_test_with_activerecord_typecast.rb +21 -0
- data/test/queue_test.rb +127 -4
- data/test/rails-tests/.gitignore +2 -0
- data/test/rails-tests/rails523.sh +23 -0
- data/test/worker_test.rb +153 -35
- metadata +51 -7
data/{readme.md → README.md}
RENAMED
@@ -1,40 +1,62 @@
|
|
1
1
|
# queue_classic
|
2
|
+
A simple, efficient worker queue for Ruby & PostgreSQL.
|
2
3
|
|
3
|
-
|
4
|
-
<b>Simple, efficient worker queue for Ruby & PostgreSQL</b>
|
5
|
-
<br />
|
6
|
-
<a href="https://travis-ci.org/QueueClassic/queue_classic"><img src="http://img.shields.io/travis/QueueClassic/queue_classic/master.svg?style=flat" /></a>
|
4
|
+
Why this over something like Resque. Two reasons:
|
7
5
|
|
8
|
-
|
6
|
+
1. Your jobs can be equeued in the same transaction as other modifications to the database, and will only be processed when everything is commited. This is a hard pattern to develop around for queues done outside your database
|
7
|
+
2. Less things to run, if you don't already have Redis or a dedicated queue in your stack.
|
9
8
|
|
10
|
-
|
11
|
-
</p>
|
9
|
+
[](https://circleci.com/gh/QueueClassic/queue_classic/tree/master)
|
12
10
|
|
11
|
+
[](http://badge.fury.io/rb/queue_classic)
|
13
12
|
|
14
|
-
**IMPORTANT NOTE
|
13
|
+
**IMPORTANT NOTE: This README is representing the current work for queue_classic, which is generally the pending next version. You can find the README for other versions:**
|
15
14
|
|
16
|
-
|
15
|
+
You can always find the latest and previous releases here:
|
17
16
|
|
18
|
-
|
19
|
-
- older stable: [v2.2.3](https://github.com/QueueClassic/queue_classic/tree/v2.2.3)
|
17
|
+
https://github.com/QueueClassic/queue_classic/releases
|
20
18
|
|
19
|
+
## Other related projects
|
20
|
+
|
21
|
+
If you're interested in this project, you might also want to checkout:
|
22
|
+
|
23
|
+
* [Que](https://github.com/que-rb/que)
|
24
|
+
* [GoodJob](https://github.com/bensheldon/good_job)
|
25
|
+
* [Delayed Job](https://github.com/collectiveidea/delayed_job)
|
26
|
+
|
27
|
+
For a list of other queues (which may or may not be Postgres backed), checkout - https://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
|
21
28
|
|
22
29
|
## What is queue_classic?
|
23
30
|
|
24
31
|
queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.
|
25
32
|
|
33
|
+
A major benefit is the ability to enqueue inside transactions, ensuring things are done only when your changes are commited.
|
34
|
+
|
35
|
+
## Other related projects
|
36
|
+
|
37
|
+
* [Queue Classic Plus](https://github.com/rainforestapp/queue_classic_plus) - adds support for retrying with specific exceptions, transaction processing of jobs, metric collection, etc
|
38
|
+
* [Queue Classic Admin](https://github.com/QueueClassic/queue_classic_admin) - Admin interface for queue_classic
|
39
|
+
* [Queue Classic Matchers](https://github.com/rainforestapp/queue_classic_matchers) - RSpec matchers for queue_classic
|
40
|
+
|
26
41
|
## Features
|
27
42
|
|
28
|
-
* Leverage of PostgreSQL's listen/notify
|
43
|
+
* Leverage of PostgreSQL's listen/notify, skip locked, and row locking.
|
29
44
|
* Support for multiple queues with heterogeneous workers.
|
30
45
|
* JSON data format.
|
31
|
-
* Forking workers.
|
32
46
|
* Workers can work multiple queues.
|
33
|
-
*
|
47
|
+
* ~~Forking workers.~~ (currently apparently broken https://github.com/QueueClassic/queue_classic/issues/207, a WIP fix exists but isn't merged tested yet https://github.com/QueueClassic/queue_classic/pull/216)
|
48
|
+
|
49
|
+
### Requirements
|
34
50
|
|
35
|
-
|
51
|
+
For this version, the requirements are as follows:
|
52
|
+
* Ruby 2.4, 2.5, 2.6 or 2.7
|
53
|
+
* Postgres ~> 9.6
|
54
|
+
* Rubygem: pg ~> 0.17
|
36
55
|
|
37
|
-
|
56
|
+
|
57
|
+
## Table of contents
|
58
|
+
|
59
|
+
* [Documentation](https://www.rubydoc.info/gems/queue_classic/)
|
38
60
|
* [Usage](#usage)
|
39
61
|
* [Setup](#setup)
|
40
62
|
* [Upgrade from earlier versions to V3.1](#upgrade-from-earlier-versions)
|
@@ -47,14 +69,14 @@ queue_classic provides a simple interface to a PostgreSQL-backed message queue.
|
|
47
69
|
|
48
70
|
## Usage
|
49
71
|
|
50
|
-
There are 2 ways to use queue_classic
|
72
|
+
There are 2 ways to use queue_classic:
|
51
73
|
|
52
74
|
* Producing Jobs
|
53
75
|
* Working Jobs
|
54
76
|
|
55
77
|
### Producing Jobs
|
56
78
|
|
57
|
-
The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method
|
79
|
+
The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method defined by the first argument. The set of arguments will be encoded as JSON and stored in the database.
|
58
80
|
|
59
81
|
```ruby
|
60
82
|
# This method has no arguments.
|
@@ -77,13 +99,13 @@ p_queue = QC::Queue.new("priority_queue")
|
|
77
99
|
p_queue.enqueue("Kernel.puts", ["hello", "world"])
|
78
100
|
```
|
79
101
|
|
80
|
-
There is also the
|
102
|
+
There is also the ability to schedule a job to run at a specified time in the future. The job will become processable after the specified time, and will be processed as-soon-as-possible.
|
81
103
|
|
82
104
|
```ruby
|
83
|
-
#
|
105
|
+
# Specify the job execution time exactly
|
84
106
|
QC.enqueue_at(Time.new(2024,01,02,10,00), "Kernel.puts", "hello future")
|
85
107
|
|
86
|
-
#
|
108
|
+
# Specify the job execution time as an offset in seconds
|
87
109
|
QC.enqueue_in(60, "Kernel.puts", "hello from 1 minute later")
|
88
110
|
```
|
89
111
|
|
@@ -93,28 +115,33 @@ There are two ways to work jobs. The first approach is to use the Rake task. The
|
|
93
115
|
|
94
116
|
#### Rake Task
|
95
117
|
|
96
|
-
Require queue_classic in your Rakefile
|
118
|
+
Require queue_classic in your Rakefile:
|
97
119
|
|
98
120
|
```ruby
|
99
121
|
require 'queue_classic'
|
100
122
|
require 'queue_classic/tasks'
|
101
123
|
```
|
102
124
|
|
103
|
-
|
125
|
+
##### Work all queues
|
126
|
+
Start the worker via the Rakefile:
|
127
|
+
|
104
128
|
```bash
|
105
129
|
$ bundle exec rake qc:work
|
106
130
|
```
|
107
131
|
|
108
|
-
|
132
|
+
##### Work a single specific queue
|
133
|
+
Setup a worker to work only a specific, non-default queue:
|
134
|
+
|
109
135
|
```bash
|
110
136
|
$ QUEUE="priority_queue" bundle exec rake qc:work
|
111
137
|
```
|
112
138
|
|
113
|
-
|
139
|
+
##### Work multiple queues
|
140
|
+
In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
|
141
|
+
|
114
142
|
```bash
|
115
143
|
$ QUEUES="priority_queue,secondary_queue" bundle exec rake qc:work
|
116
144
|
```
|
117
|
-
In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
|
118
145
|
|
119
146
|
#### Custom Worker
|
120
147
|
|
@@ -127,6 +154,23 @@ require 'queue_classic'
|
|
127
154
|
FailedQueue = QC::Queue.new("failed_jobs")
|
128
155
|
|
129
156
|
class MyWorker < QC::Worker
|
157
|
+
# A job is a Hash containing these attributes:
|
158
|
+
# :id Integer, the job id
|
159
|
+
# :method String, containing the object and method
|
160
|
+
# :args String, the arguments
|
161
|
+
# :q_name String, the queue name
|
162
|
+
# :scheduled_at Time, the scheduled time if the job was scheduled
|
163
|
+
|
164
|
+
# Execute the job using the methods and arguments
|
165
|
+
def call(job)
|
166
|
+
# Do something with the job
|
167
|
+
...
|
168
|
+
end
|
169
|
+
|
170
|
+
# This method will be called when an exception
|
171
|
+
# is raised during the execution of the job.
|
172
|
+
# First argument is the job that failed.
|
173
|
+
# Second argument is the exception.
|
130
174
|
def handle_failure(job, e)
|
131
175
|
FailedQueue.enqueue(job[:method], *job[:args])
|
132
176
|
end
|
@@ -134,18 +178,24 @@ end
|
|
134
178
|
|
135
179
|
worker = MyWorker.new
|
136
180
|
|
137
|
-
trap('INT') {exit}
|
138
|
-
trap('TERM') {worker.stop}
|
181
|
+
trap('INT') { exit }
|
182
|
+
trap('TERM') { worker.stop }
|
139
183
|
|
140
184
|
loop do
|
141
|
-
job = worker.lock_job
|
142
|
-
Timeout::timeout(5) { worker.process(job) }
|
185
|
+
queue, job = worker.lock_job
|
186
|
+
Timeout::timeout(5) { worker.process(queue, job) }
|
143
187
|
end
|
144
188
|
```
|
145
189
|
|
190
|
+
The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to inject your own worker class:
|
191
|
+
|
192
|
+
```ruby
|
193
|
+
QC.default_worker_class = MyWorker
|
194
|
+
```
|
195
|
+
|
146
196
|
## Setup
|
147
197
|
|
148
|
-
In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `
|
198
|
+
In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `psql` or use a database migration script.
|
149
199
|
|
150
200
|
### Quick Start
|
151
201
|
|
@@ -160,23 +210,37 @@ $ ruby -r queue_classic -e "QC::Worker.new.work"
|
|
160
210
|
|
161
211
|
### Ruby on Rails Setup
|
162
212
|
|
163
|
-
Declare dependencies in Gemfile
|
213
|
+
Declare dependencies in Gemfile:
|
214
|
+
|
164
215
|
```ruby
|
165
|
-
source
|
166
|
-
gem
|
216
|
+
source 'https://rubygems.org' do
|
217
|
+
gem 'queue_classic'
|
218
|
+
end
|
167
219
|
```
|
168
220
|
|
169
|
-
|
170
|
-
|
221
|
+
Install queue_classic, which adds the needed migrations for the database tables and stored procedures:
|
222
|
+
|
223
|
+
```bash
|
171
224
|
rails generate queue_classic:install
|
172
|
-
rake db:migrate
|
225
|
+
bundle exec rake db:migrate
|
173
226
|
```
|
174
227
|
|
175
|
-
|
228
|
+
#### Database connection
|
229
|
+
|
230
|
+
Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used. If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
|
231
|
+
|
232
|
+
**Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
|
176
233
|
|
177
|
-
|
234
|
+
#### Active Job
|
178
235
|
|
179
|
-
|
236
|
+
If you use Rails 4.2+ and want to use Active Job, all you need to do is to set `config.active_job.queue_adapter = :queue_classic` in your `application.rb`. Everything else will be taken care for you. You can now use the Active Job functionality from now.
|
237
|
+
|
238
|
+
Just for your information, queue_classic detects your database connection and uses it.
|
239
|
+
|
240
|
+
|
241
|
+
### Plain Ruby Setup
|
242
|
+
|
243
|
+
If you're not using rails, you can use the Rake task to prepare your database:
|
180
244
|
|
181
245
|
```bash
|
182
246
|
# Creating the table and functions
|
@@ -186,35 +250,26 @@ $ bundle exec rake qc:create
|
|
186
250
|
$ bundle exec rake qc:drop
|
187
251
|
```
|
188
252
|
|
189
|
-
|
190
|
-
|
191
|
-
#### Ruby on Rails
|
192
|
-
|
193
|
-
Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used.
|
194
|
-
|
195
|
-
If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
|
253
|
+
#### Database connection
|
196
254
|
|
197
|
-
|
255
|
+
By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
|
198
256
|
|
257
|
+
## Upgrading from earlier versions
|
199
258
|
|
200
|
-
#### Other Ruby apps
|
201
|
-
|
202
|
-
By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
|
203
|
-
|
204
|
-
## Upgrade from earlier versions
|
205
259
|
If you are upgrading from a previous version of queue_classic, you might need some new database columns and/or functions. Luckily enough for you, it is easy to do so.
|
206
260
|
|
207
261
|
### Ruby on Rails
|
208
262
|
|
209
|
-
|
263
|
+
These two commands will add the newer migrations:
|
210
264
|
|
211
|
-
```
|
265
|
+
```bash
|
212
266
|
rails generate queue_classic:install
|
213
|
-
rake db:migrate
|
267
|
+
bundle exec rake db:migrate
|
214
268
|
```
|
269
|
+
|
215
270
|
### Rake Task
|
216
271
|
|
217
|
-
This rake task will
|
272
|
+
This rake task will update you to the latest version:
|
218
273
|
```bash
|
219
274
|
# Updating the table and functions
|
220
275
|
$ bundle exec rake qc:update
|
@@ -224,29 +279,24 @@ $ bundle exec rake qc:update
|
|
224
279
|
|
225
280
|
All configuration takes place in the form of environment vars. See [queue_classic.rb](https://github.com/QueueClassic/queue_classic/blob/master/lib/queue_classic.rb#L23-62) for a list of options.
|
226
281
|
|
227
|
-
## JSON
|
228
|
-
|
229
|
-
If you are running PostgreSQL 9.2 or higher, queue_classic will use the [json](http://www.postgresql.org/docs/9.2/static/datatype-json.html) datatype for storing arguments. Versions 9.1 and lower will use the 'text' column. If you have installed queue_classic prior to version 2.1.4 and are running PostgreSQL >= 9.2, run the following to switch to using the json type:
|
230
|
-
```
|
231
|
-
alter table queue_classic_jobs alter column args type json using (args::json);
|
232
|
-
```
|
233
|
-
|
234
282
|
## Logging
|
235
283
|
|
236
284
|
By default queue_classic does not talk very much.
|
237
285
|
If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: DEBUG and MEASURE.
|
238
286
|
|
239
287
|
### Measure
|
240
|
-
This will output the time to process and that kind of thing. To enable it, set the `QC_MEASURE`:
|
241
288
|
|
242
|
-
|
289
|
+
This will output the time to process and some more statistics. To enable it, set the `QC_MEASURE`:
|
290
|
+
|
291
|
+
```bash
|
243
292
|
export QC_MEASURE="true"
|
244
293
|
```
|
245
294
|
|
246
295
|
### Debug
|
296
|
+
|
247
297
|
You can enable the debug output by setting the `DEBUG` environment variable:
|
248
298
|
|
249
|
-
```
|
299
|
+
```bash
|
250
300
|
export DEBUG="true"
|
251
301
|
```
|
252
302
|
|
@@ -254,34 +304,21 @@ export DEBUG="true"
|
|
254
304
|
|
255
305
|
If you think you have found a bug, feel free to open an issue. Use the following template for the new issue:
|
256
306
|
|
257
|
-
1. List
|
307
|
+
1. List your versions: Ruby, PostgreSQL, queue_classic.
|
258
308
|
2. Define what you would have expected to happen.
|
259
309
|
3. List what actually happened.
|
260
310
|
4. Provide sample codes & commands which will reproduce the problem.
|
261
311
|
|
262
|
-
If you have general questions about how to use queue_classic, send a message to the mailing list:
|
263
|
-
|
264
|
-
https://groups.google.com/d/forum/queue_classic
|
265
|
-
|
266
312
|
## Hacking on queue_classic
|
267
313
|
|
268
|
-
[](https://travis-ci.org/QueueClassic/queue_classic)
|
269
|
-
[](https://codeclimate.com/github/QueueClassic/queue_classic)
|
270
|
-
|
271
|
-
### Dependencies
|
272
|
-
|
273
|
-
* Ruby 1.9.2 (tests work in 1.8.7 but compatibility is not guaranteed or supported)
|
274
|
-
* Postgres ~> 9.0
|
275
|
-
* Rubygem: pg ~> 0.11.0
|
276
|
-
* For JRuby, see [queue_classic_java](https://github.com/bdon/queue_classic_java)
|
277
|
-
|
278
314
|
### Running Tests
|
279
315
|
|
280
316
|
```bash
|
281
317
|
$ bundle
|
282
318
|
$ createdb queue_classic_test
|
283
319
|
$ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
|
284
|
-
$ rake
|
320
|
+
$ bundle exec rake # run all tests
|
321
|
+
$ bundle exec ruby test/queue_test.rb # run a single test
|
285
322
|
```
|
286
323
|
|
287
324
|
## License
|
data/Rakefile
ADDED
@@ -0,0 +1,16 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
$:.unshift("lib")
|
4
|
+
|
5
|
+
require "bundler/gem_tasks"
|
6
|
+
require "rake/testtask"
|
7
|
+
require "./lib/queue_classic"
|
8
|
+
require "./lib/queue_classic/tasks"
|
9
|
+
|
10
|
+
task :default => ['test']
|
11
|
+
Rake::TestTask.new do |t|
|
12
|
+
t.libs << 'test'
|
13
|
+
t.test_files = FileList['test/**/*_test.rb']
|
14
|
+
t.verbose = true
|
15
|
+
t.warning = true
|
16
|
+
end
|
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require 'rails/generators'
|
2
4
|
require 'rails/generators/migration'
|
3
5
|
require 'active_record'
|
@@ -31,6 +33,10 @@ module QC
|
|
31
33
|
if self.class.migration_exists?('db/migrate', 'update_queue_classic_3_1_0').nil?
|
32
34
|
migration_template 'update_queue_classic_3_1_0.rb', 'db/migrate/update_queue_classic_3_1_0.rb'
|
33
35
|
end
|
36
|
+
|
37
|
+
if self.class.migration_exists?('db/migrate', 'update_queue_classic_4_0_0').nil?
|
38
|
+
migration_template 'update_queue_classic_4_0_0.rb', 'db/migrate/update_queue_classic_4_0_0.rb'
|
39
|
+
end
|
34
40
|
end
|
35
41
|
end
|
36
42
|
end
|
@@ -0,0 +1,86 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module QC
|
4
|
+
module Config
|
5
|
+
# You can use the APP_NAME to query for
|
6
|
+
# postgres related process information in the
|
7
|
+
# pg_stat_activity table.
|
8
|
+
def app_name
|
9
|
+
@app_name ||= ENV["QC_APP_NAME"] || "queue_classic"
|
10
|
+
end
|
11
|
+
|
12
|
+
# Number of seconds to block on the listen chanel for new jobs.
|
13
|
+
def wait_time
|
14
|
+
@wait_time ||= (ENV["QC_LISTEN_TIME"] || 5).to_i
|
15
|
+
end
|
16
|
+
|
17
|
+
# Why do you want to change the table name?
|
18
|
+
# Just deal with the default OK?
|
19
|
+
# If you do want to change this, you will
|
20
|
+
# need to update the PL/pgSQL lock_head() function.
|
21
|
+
# Come on. Don't do it.... Just stick with the default.
|
22
|
+
def table_name
|
23
|
+
@table_name ||= "queue_classic_jobs"
|
24
|
+
end
|
25
|
+
|
26
|
+
def queue
|
27
|
+
@queue = ENV["QUEUE"] || "default"
|
28
|
+
end
|
29
|
+
|
30
|
+
# The default queue used by `QC.enqueue`.
|
31
|
+
def default_queue
|
32
|
+
@default_queue ||= Queue.new(QC.queue)
|
33
|
+
end
|
34
|
+
|
35
|
+
def default_queue=(queue)
|
36
|
+
@default_queue = queue
|
37
|
+
end
|
38
|
+
|
39
|
+
# Each row in the table will have a column that
|
40
|
+
# notes the queue. You can point your workers
|
41
|
+
# at different queues.
|
42
|
+
def queues
|
43
|
+
@queues ||= (ENV["QUEUES"] && ENV["QUEUES"].split(",").map(&:strip)) || []
|
44
|
+
end
|
45
|
+
|
46
|
+
# Set this to 1 for strict FIFO.
|
47
|
+
# There is nothing special about 9....
|
48
|
+
def top_bound
|
49
|
+
@top_bound ||= (ENV["QC_TOP_BOUND"] || 9).to_i
|
50
|
+
end
|
51
|
+
|
52
|
+
# Set this variable if you wish for
|
53
|
+
# the worker to fork a UNIX process for
|
54
|
+
# each locked job. Remember to re-establish
|
55
|
+
# any database connections. See the worker
|
56
|
+
# for more details.
|
57
|
+
def fork_worker?
|
58
|
+
@fork_worker ||= (!ENV["QC_FORK_WORKER"].nil?)
|
59
|
+
end
|
60
|
+
|
61
|
+
# The worker class instantiated by QC's rake tasks.
|
62
|
+
def default_worker_class
|
63
|
+
@worker_class ||= (ENV["QC_DEFAULT_WORKER_CLASS"] && Kernel.const_get(ENV["QC_DEFAULT_WORKER_CLASS"]) ||
|
64
|
+
QC::Worker)
|
65
|
+
|
66
|
+
end
|
67
|
+
|
68
|
+
def default_worker_class=(worker_class)
|
69
|
+
@worker_class = worker_class
|
70
|
+
end
|
71
|
+
|
72
|
+
# reset memoized configuration
|
73
|
+
def reset_config
|
74
|
+
# TODO: we might want to think about storing these in a Hash.
|
75
|
+
@app_name = nil
|
76
|
+
@wait_time = nil
|
77
|
+
@table_name = nil
|
78
|
+
@queue = nil
|
79
|
+
@default_queue = nil
|
80
|
+
@queues = nil
|
81
|
+
@top_bound = nil
|
82
|
+
@fork_worker = nil
|
83
|
+
@worker_class = nil
|
84
|
+
end
|
85
|
+
end
|
86
|
+
end
|
@@ -1,27 +1,37 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
require 'uri'
|
2
4
|
require 'pg'
|
3
5
|
|
4
6
|
module QC
|
5
7
|
class ConnAdapter
|
6
8
|
|
7
|
-
|
8
|
-
|
9
|
-
@
|
9
|
+
def initialize(args={})
|
10
|
+
@active_record_connection_share = args[:active_record_connection_share]
|
11
|
+
@_connection = args[:connection]
|
10
12
|
@mutex = Mutex.new
|
11
13
|
end
|
12
14
|
|
15
|
+
def connection
|
16
|
+
if @active_record_connection_share && Object.const_defined?('ActiveRecord')
|
17
|
+
ActiveRecord::Base.connection.raw_connection
|
18
|
+
else
|
19
|
+
@_connection ||= establish_new
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
13
23
|
def execute(stmt, *params)
|
14
24
|
@mutex.synchronize do
|
15
25
|
QC.log(:at => "exec_sql", :sql => stmt.inspect)
|
16
26
|
begin
|
17
27
|
params = nil if params.empty?
|
18
|
-
r =
|
28
|
+
r = connection.exec(stmt, params)
|
19
29
|
result = []
|
20
30
|
r.each {|t| result << t}
|
21
31
|
result.length > 1 ? result : result.pop
|
22
|
-
rescue
|
32
|
+
rescue PG::Error => e
|
23
33
|
QC.log(:error => e.inspect)
|
24
|
-
|
34
|
+
connection.reset
|
25
35
|
raise
|
26
36
|
end
|
27
37
|
end
|
@@ -30,10 +40,10 @@ module QC
|
|
30
40
|
def wait(time, *channels)
|
31
41
|
@mutex.synchronize do
|
32
42
|
listen_cmds = channels.map {|c| 'LISTEN "' + c.to_s + '"'}
|
33
|
-
|
43
|
+
connection.exec(listen_cmds.join(';'))
|
34
44
|
wait_for_notify(time)
|
35
45
|
unlisten_cmds = channels.map {|c| 'UNLISTEN "' + c.to_s + '"'}
|
36
|
-
|
46
|
+
connection.exec(unlisten_cmds.join(';'))
|
37
47
|
drain_notify
|
38
48
|
end
|
39
49
|
end
|
@@ -41,23 +51,30 @@ module QC
|
|
41
51
|
def disconnect
|
42
52
|
@mutex.synchronize do
|
43
53
|
begin
|
44
|
-
|
54
|
+
connection.close
|
45
55
|
rescue => e
|
46
56
|
QC.log(:at => 'disconnect', :error => e.message)
|
47
57
|
end
|
48
58
|
end
|
49
59
|
end
|
50
60
|
|
61
|
+
def server_version
|
62
|
+
@server_version ||= begin
|
63
|
+
version = execute("SHOW server_version_num;")["server_version_num"]
|
64
|
+
version && version.to_i
|
65
|
+
end
|
66
|
+
end
|
67
|
+
|
51
68
|
private
|
52
69
|
|
53
70
|
def wait_for_notify(t)
|
54
71
|
Array.new.tap do |msgs|
|
55
|
-
|
72
|
+
connection.wait_for_notify(t) {|event, pid, msg| msgs << msg}
|
56
73
|
end
|
57
74
|
end
|
58
75
|
|
59
76
|
def drain_notify
|
60
|
-
until
|
77
|
+
until connection.notifies.nil?
|
61
78
|
QC.log(:at => "drain_notifications")
|
62
79
|
end
|
63
80
|
end
|
@@ -70,11 +87,16 @@ module QC
|
|
70
87
|
|
71
88
|
def establish_new
|
72
89
|
QC.log(:at => "establish_conn")
|
73
|
-
conn =
|
74
|
-
if conn.status !=
|
90
|
+
conn = PG.connect(*normalize_db_url(db_url))
|
91
|
+
if conn.status != PG::CONNECTION_OK
|
75
92
|
QC.log(:error => conn.error)
|
76
93
|
end
|
77
|
-
|
94
|
+
|
95
|
+
if conn.server_version < 90600
|
96
|
+
raise "This version of Queue Classic does not support Postgres older than 9.6 (90600). This version is #{conn.server_version}. If you need that support, please use an older version."
|
97
|
+
end
|
98
|
+
|
99
|
+
conn.exec("SET application_name = '#{QC.app_name}'")
|
78
100
|
conn
|
79
101
|
end
|
80
102
|
|
@@ -85,7 +107,7 @@ module QC
|
|
85
107
|
[
|
86
108
|
host, # host or percent-encoded socket path
|
87
109
|
url.port || 5432,
|
88
|
-
nil,
|
110
|
+
nil, nil, #opts, tty
|
89
111
|
url.path.gsub("/",""), # database name
|
90
112
|
url.user,
|
91
113
|
url.password
|
@@ -99,6 +121,5 @@ module QC
|
|
99
121
|
raise(ArgumentError, "missing QC_DATABASE_URL or DATABASE_URL")
|
100
122
|
@db_url = URI.parse(url)
|
101
123
|
end
|
102
|
-
|
103
124
|
end
|
104
125
|
end
|