queue_classic 3.1.0.RC1 → 4.0.0.pre.beta1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (44) hide show
  1. checksums.yaml +5 -5
  2. data/.circleci/config.yml +192 -0
  3. data/.gitignore +11 -0
  4. data/CHANGELOG.md +192 -0
  5. data/CODE_OF_CONDUCT.md +46 -0
  6. data/CONTRIBUTING.md +17 -0
  7. data/Gemfile +12 -0
  8. data/LICENSE.txt +20 -0
  9. data/{readme.md → README.md} +120 -83
  10. data/Rakefile +16 -0
  11. data/lib/generators/queue_classic/install_generator.rb +6 -0
  12. data/lib/generators/queue_classic/templates/add_queue_classic.rb +3 -1
  13. data/lib/generators/queue_classic/templates/update_queue_classic_3_0_0.rb +3 -1
  14. data/lib/generators/queue_classic/templates/update_queue_classic_3_0_2.rb +3 -1
  15. data/lib/generators/queue_classic/templates/update_queue_classic_3_1_0.rb +3 -1
  16. data/lib/generators/queue_classic/templates/update_queue_classic_4_0_0.rb +11 -0
  17. data/lib/queue_classic/config.rb +86 -0
  18. data/lib/queue_classic/conn_adapter.rb +37 -16
  19. data/lib/queue_classic/queue.rb +76 -18
  20. data/lib/queue_classic/railtie.rb +2 -0
  21. data/lib/queue_classic/setup.rb +24 -7
  22. data/lib/queue_classic/tasks.rb +7 -8
  23. data/lib/queue_classic/version.rb +5 -0
  24. data/lib/queue_classic/worker.rb +18 -12
  25. data/lib/queue_classic.rb +50 -58
  26. data/queue_classic.gemspec +25 -0
  27. data/sql/create_table.sql +7 -14
  28. data/sql/ddl.sql +6 -82
  29. data/sql/downgrade_from_4_0_0.sql +88 -0
  30. data/sql/update_to_3_0_0.sql +5 -5
  31. data/sql/update_to_3_1_0.sql +6 -6
  32. data/sql/update_to_4_0_0.sql +6 -0
  33. data/test/benchmark_test.rb +15 -12
  34. data/test/config_test.rb +123 -0
  35. data/test/helper.rb +47 -3
  36. data/test/helper.sql +25 -0
  37. data/test/lib/queue_classic_rails_connection_test.rb +16 -10
  38. data/test/lib/queue_classic_test.rb +15 -3
  39. data/test/lib/queue_classic_test_with_activerecord_typecast.rb +21 -0
  40. data/test/queue_test.rb +127 -4
  41. data/test/rails-tests/.gitignore +2 -0
  42. data/test/rails-tests/rails523.sh +23 -0
  43. data/test/worker_test.rb +153 -35
  44. metadata +51 -7
@@ -1,40 +1,62 @@
1
1
  # queue_classic
2
+ A simple, efficient worker queue for Ruby & PostgreSQL.
2
3
 
3
- <p align="center">
4
- <b>Simple, efficient worker queue for Ruby & PostgreSQL</b>
5
- <br />
6
- <a href="https://travis-ci.org/QueueClassic/queue_classic"><img src="http://img.shields.io/travis/QueueClassic/queue_classic/master.svg?style=flat" /></a>
4
+ Why this over something like Resque. Two reasons:
7
5
 
8
- <a href="https://codeclimate.com/github/QueueClassic/queue_classic"><img src="http://img.shields.io/codeclimate/github/QueueClassic/queue_classic.svg?style=flat" /></a>
6
+ 1. Your jobs can be equeued in the same transaction as other modifications to the database, and will only be processed when everything is commited. This is a hard pattern to develop around for queues done outside your database
7
+ 2. Less things to run, if you don't already have Redis or a dedicated queue in your stack.
9
8
 
10
- <a href="http://badge.fury.io/rb/queue_classic"><img src="http://img.shields.io/gem/v/queue_classic.svg?style=flat" alt="Gem Version" height="18"></a>
11
- </p>
9
+ [![CircleCI](https://circleci.com/gh/QueueClassic/queue_classic/tree/master.svg?style=svg)](https://circleci.com/gh/QueueClassic/queue_classic/tree/master)
12
10
 
11
+ [![Gem Version](http://img.shields.io/gem/v/queue_classic.svg?style=flat)](http://badge.fury.io/rb/queue_classic)
13
12
 
14
- **IMPORTANT NOTE REGARDING VERSIONS**
13
+ **IMPORTANT NOTE: This README is representing the current work for queue_classic, which is generally the pending next version. You can find the README for other versions:**
15
14
 
16
- **This README is representing the current work for queue_classic 3.1, which is neither complete or stable. You can find the README for stable versions:**
15
+ You can always find the latest and previous releases here:
17
16
 
18
- - latest stable can be found: [v3.0.X](https://github.com/QueueClassic/queue_classic/tree/3-0-stable)
19
- - older stable: [v2.2.3](https://github.com/QueueClassic/queue_classic/tree/v2.2.3)
17
+ https://github.com/QueueClassic/queue_classic/releases
20
18
 
19
+ ## Other related projects
20
+
21
+ If you're interested in this project, you might also want to checkout:
22
+
23
+ * [Que](https://github.com/que-rb/que)
24
+ * [GoodJob](https://github.com/bensheldon/good_job)
25
+ * [Delayed Job](https://github.com/collectiveidea/delayed_job)
26
+
27
+ For a list of other queues (which may or may not be Postgres backed), checkout - https://edgeapi.rubyonrails.org/classes/ActiveJob/QueueAdapters.html
21
28
 
22
29
  ## What is queue_classic?
23
30
 
24
31
  queue_classic provides a simple interface to a PostgreSQL-backed message queue. queue_classic specializes in concurrent locking and minimizing database load while providing a simple, intuitive developer experience. queue_classic assumes that you are already using PostgreSQL in your production environment and that adding another dependency (e.g. redis, beanstalkd, 0mq) is undesirable.
25
32
 
33
+ A major benefit is the ability to enqueue inside transactions, ensuring things are done only when your changes are commited.
34
+
35
+ ## Other related projects
36
+
37
+ * [Queue Classic Plus](https://github.com/rainforestapp/queue_classic_plus) - adds support for retrying with specific exceptions, transaction processing of jobs, metric collection, etc
38
+ * [Queue Classic Admin](https://github.com/QueueClassic/queue_classic_admin) - Admin interface for queue_classic
39
+ * [Queue Classic Matchers](https://github.com/rainforestapp/queue_classic_matchers) - RSpec matchers for queue_classic
40
+
26
41
  ## Features
27
42
 
28
- * Leverage of PostgreSQL's listen/notify & row locking.
43
+ * Leverage of PostgreSQL's listen/notify, skip locked, and row locking.
29
44
  * Support for multiple queues with heterogeneous workers.
30
45
  * JSON data format.
31
- * Forking workers.
32
46
  * Workers can work multiple queues.
33
- * Reduced row contention using a [relaxed FIFO](http://www.cs.tau.ac.il/~shanir/nir-pubs-web/Papers/Lock_Free.pdf) technique.
47
+ * ~~Forking workers.~~ (currently apparently broken https://github.com/QueueClassic/queue_classic/issues/207, a WIP fix exists but isn't merged tested yet https://github.com/QueueClassic/queue_classic/pull/216)
48
+
49
+ ### Requirements
34
50
 
35
- ## Table of content
51
+ For this version, the requirements are as follows:
52
+ * Ruby 2.4, 2.5, 2.6 or 2.7
53
+ * Postgres ~> 9.6
54
+ * Rubygem: pg ~> 0.17
36
55
 
37
- * [Documentation](http://rubydoc.info/gems/queue_classic/2.2.3/frames)
56
+
57
+ ## Table of contents
58
+
59
+ * [Documentation](https://www.rubydoc.info/gems/queue_classic/)
38
60
  * [Usage](#usage)
39
61
  * [Setup](#setup)
40
62
  * [Upgrade from earlier versions to V3.1](#upgrade-from-earlier-versions)
@@ -47,14 +69,14 @@ queue_classic provides a simple interface to a PostgreSQL-backed message queue.
47
69
 
48
70
  ## Usage
49
71
 
50
- There are 2 ways to use queue_classic.
72
+ There are 2 ways to use queue_classic:
51
73
 
52
74
  * Producing Jobs
53
75
  * Working Jobs
54
76
 
55
77
  ### Producing Jobs
56
78
 
57
- The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method invocation defined by the first argument. The set of arguments will be encoded as JSON in the database.
79
+ The first argument is a string which represents a ruby object and a method name. The second argument(s) will be passed along as arguments to the method defined by the first argument. The set of arguments will be encoded as JSON and stored in the database.
58
80
 
59
81
  ```ruby
60
82
  # This method has no arguments.
@@ -77,13 +99,13 @@ p_queue = QC::Queue.new("priority_queue")
77
99
  p_queue.enqueue("Kernel.puts", ["hello", "world"])
78
100
  ```
79
101
 
80
- There is also the possibility to schedule a job at a specified time in the future. It will not be worked off before that specified time.
102
+ There is also the ability to schedule a job to run at a specified time in the future. The job will become processable after the specified time, and will be processed as-soon-as-possible.
81
103
 
82
104
  ```ruby
83
- # Specifying the job execution time exactly.
105
+ # Specify the job execution time exactly
84
106
  QC.enqueue_at(Time.new(2024,01,02,10,00), "Kernel.puts", "hello future")
85
107
 
86
- # Specifying the job execution time as an offset in seconds.
108
+ # Specify the job execution time as an offset in seconds
87
109
  QC.enqueue_in(60, "Kernel.puts", "hello from 1 minute later")
88
110
  ```
89
111
 
@@ -93,28 +115,33 @@ There are two ways to work jobs. The first approach is to use the Rake task. The
93
115
 
94
116
  #### Rake Task
95
117
 
96
- Require queue_classic in your Rakefile.
118
+ Require queue_classic in your Rakefile:
97
119
 
98
120
  ```ruby
99
121
  require 'queue_classic'
100
122
  require 'queue_classic/tasks'
101
123
  ```
102
124
 
103
- Start the worker via the Rakefile.
125
+ ##### Work all queues
126
+ Start the worker via the Rakefile:
127
+
104
128
  ```bash
105
129
  $ bundle exec rake qc:work
106
130
  ```
107
131
 
108
- Setup a worker to work a non-default queue.
132
+ ##### Work a single specific queue
133
+ Setup a worker to work only a specific, non-default queue:
134
+
109
135
  ```bash
110
136
  $ QUEUE="priority_queue" bundle exec rake qc:work
111
137
  ```
112
138
 
113
- Setup a worker to work multiple queues.
139
+ ##### Work multiple queues
140
+ In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
141
+
114
142
  ```bash
115
143
  $ QUEUES="priority_queue,secondary_queue" bundle exec rake qc:work
116
144
  ```
117
- In this scenario, on each iteration of the worker's loop, it will look for jobs in the first queue prior to looking at the second queue. This means that the first queue must be empty before the worker will look at the second queue.
118
145
 
119
146
  #### Custom Worker
120
147
 
@@ -127,6 +154,23 @@ require 'queue_classic'
127
154
  FailedQueue = QC::Queue.new("failed_jobs")
128
155
 
129
156
  class MyWorker < QC::Worker
157
+ # A job is a Hash containing these attributes:
158
+ # :id Integer, the job id
159
+ # :method String, containing the object and method
160
+ # :args String, the arguments
161
+ # :q_name String, the queue name
162
+ # :scheduled_at Time, the scheduled time if the job was scheduled
163
+
164
+ # Execute the job using the methods and arguments
165
+ def call(job)
166
+ # Do something with the job
167
+ ...
168
+ end
169
+
170
+ # This method will be called when an exception
171
+ # is raised during the execution of the job.
172
+ # First argument is the job that failed.
173
+ # Second argument is the exception.
130
174
  def handle_failure(job, e)
131
175
  FailedQueue.enqueue(job[:method], *job[:args])
132
176
  end
@@ -134,18 +178,24 @@ end
134
178
 
135
179
  worker = MyWorker.new
136
180
 
137
- trap('INT') {exit}
138
- trap('TERM') {worker.stop}
181
+ trap('INT') { exit }
182
+ trap('TERM') { worker.stop }
139
183
 
140
184
  loop do
141
- job = worker.lock_job
142
- Timeout::timeout(5) { worker.process(job) }
185
+ queue, job = worker.lock_job
186
+ Timeout::timeout(5) { worker.process(queue, job) }
143
187
  end
144
188
  ```
145
189
 
190
+ The `qc:work` rake task uses `QC::Worker` by default. However, it's easy to inject your own worker class:
191
+
192
+ ```ruby
193
+ QC.default_worker_class = MyWorker
194
+ ```
195
+
146
196
  ## Setup
147
197
 
148
- In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `PSQL(1)` or use a database migration script.
198
+ In addition to installing the rubygem, you will need to prepare your database. Database preparation includes creating a table and loading PL/pgSQL functions. You can issue the database preparation commands using `psql` or use a database migration script.
149
199
 
150
200
  ### Quick Start
151
201
 
@@ -160,23 +210,37 @@ $ ruby -r queue_classic -e "QC::Worker.new.work"
160
210
 
161
211
  ### Ruby on Rails Setup
162
212
 
163
- Declare dependencies in Gemfile.
213
+ Declare dependencies in Gemfile:
214
+
164
215
  ```ruby
165
- source "http://rubygems.org"
166
- gem "queue_classic", "~> 3.0.0"
216
+ source 'https://rubygems.org' do
217
+ gem 'queue_classic'
218
+ end
167
219
  ```
168
220
 
169
- Add the database tables and stored procedures.
170
- ```
221
+ Install queue_classic, which adds the needed migrations for the database tables and stored procedures:
222
+
223
+ ```bash
171
224
  rails generate queue_classic:install
172
- rake db:migrate
225
+ bundle exec rake db:migrate
173
226
  ```
174
227
 
175
- If you want to use queue_classic with Active Job (Rails 4.2+), you need to set `Rails.application.config.active_job.queue_adapter = :queue_classic`. Everything else will be taken care for you, just use the Active Job API.
228
+ #### Database connection
229
+
230
+ Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used. If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
231
+
232
+ **Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
176
233
 
177
- ### Rake Task Setup
234
+ #### Active Job
178
235
 
179
- Alternatively, you can use the Rake task to prepare your database.
236
+ If you use Rails 4.2+ and want to use Active Job, all you need to do is to set `config.active_job.queue_adapter = :queue_classic` in your `application.rb`. Everything else will be taken care for you. You can now use the Active Job functionality from now.
237
+
238
+ Just for your information, queue_classic detects your database connection and uses it.
239
+
240
+
241
+ ### Plain Ruby Setup
242
+
243
+ If you're not using rails, you can use the Rake task to prepare your database:
180
244
 
181
245
  ```bash
182
246
  # Creating the table and functions
@@ -186,35 +250,26 @@ $ bundle exec rake qc:create
186
250
  $ bundle exec rake qc:drop
187
251
  ```
188
252
 
189
- ### Database connection
190
-
191
- #### Ruby on Rails
192
-
193
- Starting with with queue_classic 3.1, Rails is automatically detected and its connection is used.
194
-
195
- If you don't want to use the automatic database connection, set this environment variable to false: `export QC_RAILS_DATABASE=false`
253
+ #### Database connection
196
254
 
197
- **Note on using ActiveRecord migrations:** If you use the migration, and you wish to use commands that reset the database from the stored schema (e.g. `rake db:reset`), your application must be configured with `config.active_record.schema_format = :sql` in `config/application.rb`. If you don't do this, the PL/pgSQL function that queue_classic creates will be lost when you reset the database.
255
+ By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
198
256
 
257
+ ## Upgrading from earlier versions
199
258
 
200
- #### Other Ruby apps
201
-
202
- By default, queue_classic will use the QC_DATABASE_URL falling back on DATABASE_URL. The URL must be in the following format: `postgres://username:password@localhost/database_name`. If you use Heroku's PostgreSQL service, this will already be set. If you don't want to set this variable, you can set the connection in an initializer. **QueueClassic will maintain its own connection to the database.** This may double the number of connections to your database.
203
-
204
- ## Upgrade from earlier versions
205
259
  If you are upgrading from a previous version of queue_classic, you might need some new database columns and/or functions. Luckily enough for you, it is easy to do so.
206
260
 
207
261
  ### Ruby on Rails
208
262
 
209
- You just need to run those lines, which will copy the new required migrations:
263
+ These two commands will add the newer migrations:
210
264
 
211
- ```
265
+ ```bash
212
266
  rails generate queue_classic:install
213
- rake db:migrate
267
+ bundle exec rake db:migrate
214
268
  ```
269
+
215
270
  ### Rake Task
216
271
 
217
- This rake task will get you covered:
272
+ This rake task will update you to the latest version:
218
273
  ```bash
219
274
  # Updating the table and functions
220
275
  $ bundle exec rake qc:update
@@ -224,29 +279,24 @@ $ bundle exec rake qc:update
224
279
 
225
280
  All configuration takes place in the form of environment vars. See [queue_classic.rb](https://github.com/QueueClassic/queue_classic/blob/master/lib/queue_classic.rb#L23-62) for a list of options.
226
281
 
227
- ## JSON
228
-
229
- If you are running PostgreSQL 9.2 or higher, queue_classic will use the [json](http://www.postgresql.org/docs/9.2/static/datatype-json.html) datatype for storing arguments. Versions 9.1 and lower will use the 'text' column. If you have installed queue_classic prior to version 2.1.4 and are running PostgreSQL >= 9.2, run the following to switch to using the json type:
230
- ```
231
- alter table queue_classic_jobs alter column args type json using (args::json);
232
- ```
233
-
234
282
  ## Logging
235
283
 
236
284
  By default queue_classic does not talk very much.
237
285
  If you find yourself in a situation where you need to know what's happening inside QC, there are two different kind of logging you can enable: DEBUG and MEASURE.
238
286
 
239
287
  ### Measure
240
- This will output the time to process and that kind of thing. To enable it, set the `QC_MEASURE`:
241
288
 
242
- ```
289
+ This will output the time to process and some more statistics. To enable it, set the `QC_MEASURE`:
290
+
291
+ ```bash
243
292
  export QC_MEASURE="true"
244
293
  ```
245
294
 
246
295
  ### Debug
296
+
247
297
  You can enable the debug output by setting the `DEBUG` environment variable:
248
298
 
249
- ```
299
+ ```bash
250
300
  export DEBUG="true"
251
301
  ```
252
302
 
@@ -254,34 +304,21 @@ export DEBUG="true"
254
304
 
255
305
  If you think you have found a bug, feel free to open an issue. Use the following template for the new issue:
256
306
 
257
- 1. List Versions: Ruby, PostgreSQL, queue_classic.
307
+ 1. List your versions: Ruby, PostgreSQL, queue_classic.
258
308
  2. Define what you would have expected to happen.
259
309
  3. List what actually happened.
260
310
  4. Provide sample codes & commands which will reproduce the problem.
261
311
 
262
- If you have general questions about how to use queue_classic, send a message to the mailing list:
263
-
264
- https://groups.google.com/d/forum/queue_classic
265
-
266
312
  ## Hacking on queue_classic
267
313
 
268
- [![Build Status](https://travis-ci.org/QueueClassic/queue_classic.svg?branch=master)](https://travis-ci.org/QueueClassic/queue_classic)
269
- [![Code Climate](https://codeclimate.com/github/QueueClassic/queue_classic.png)](https://codeclimate.com/github/QueueClassic/queue_classic)
270
-
271
- ### Dependencies
272
-
273
- * Ruby 1.9.2 (tests work in 1.8.7 but compatibility is not guaranteed or supported)
274
- * Postgres ~> 9.0
275
- * Rubygem: pg ~> 0.11.0
276
- * For JRuby, see [queue_classic_java](https://github.com/bdon/queue_classic_java)
277
-
278
314
  ### Running Tests
279
315
 
280
316
  ```bash
281
317
  $ bundle
282
318
  $ createdb queue_classic_test
283
319
  $ export QC_DATABASE_URL="postgres://username:pass@localhost/queue_classic_test"
284
- $ rake
320
+ $ bundle exec rake # run all tests
321
+ $ bundle exec ruby test/queue_test.rb # run a single test
285
322
  ```
286
323
 
287
324
  ## License
data/Rakefile ADDED
@@ -0,0 +1,16 @@
1
+ # frozen_string_literal: true
2
+
3
+ $:.unshift("lib")
4
+
5
+ require "bundler/gem_tasks"
6
+ require "rake/testtask"
7
+ require "./lib/queue_classic"
8
+ require "./lib/queue_classic/tasks"
9
+
10
+ task :default => ['test']
11
+ Rake::TestTask.new do |t|
12
+ t.libs << 'test'
13
+ t.test_files = FileList['test/**/*_test.rb']
14
+ t.verbose = true
15
+ t.warning = true
16
+ end
@@ -1,3 +1,5 @@
1
+ # frozen_string_literal: true
2
+
1
3
  require 'rails/generators'
2
4
  require 'rails/generators/migration'
3
5
  require 'active_record'
@@ -31,6 +33,10 @@ module QC
31
33
  if self.class.migration_exists?('db/migrate', 'update_queue_classic_3_1_0').nil?
32
34
  migration_template 'update_queue_classic_3_1_0.rb', 'db/migrate/update_queue_classic_3_1_0.rb'
33
35
  end
36
+
37
+ if self.class.migration_exists?('db/migrate', 'update_queue_classic_4_0_0').nil?
38
+ migration_template 'update_queue_classic_4_0_0.rb', 'db/migrate/update_queue_classic_4_0_0.rb'
39
+ end
34
40
  end
35
41
  end
36
42
  end
@@ -1,4 +1,6 @@
1
- class AddQueueClassic < ActiveRecord::Migration
1
+ # frozen_string_literal: true
2
+
3
+ class AddQueueClassic < ActiveRecord::Migration[4.2]
2
4
  def self.up
3
5
  QC::Setup.create
4
6
  end
@@ -1,4 +1,6 @@
1
- class UpdateQueueClassic300 < ActiveRecord::Migration
1
+ # frozen_string_literal: true
2
+
3
+ class UpdateQueueClassic300 < ActiveRecord::Migration[4.2]
2
4
  def self.up
3
5
  QC::Setup.update_to_3_0_0
4
6
  end
@@ -1,4 +1,6 @@
1
- class UpdateQueueClassic302 < ActiveRecord::Migration
1
+ # frozen_string_literal: true
2
+
3
+ class UpdateQueueClassic302 < ActiveRecord::Migration[4.2]
2
4
  def self.up
3
5
  QC::Setup.update_to_3_0_0
4
6
  end
@@ -1,4 +1,6 @@
1
- class UpdateQueueClassic310 < ActiveRecord::Migration
1
+ # frozen_string_literal: true
2
+
3
+ class UpdateQueueClassic310 < ActiveRecord::Migration[4.2]
2
4
  def self.up
3
5
  QC::Setup.update_to_3_1_0
4
6
  end
@@ -0,0 +1,11 @@
1
+ # frozen_string_literal: true
2
+
3
+ class UpdateQueueClassic400 < ActiveRecord::Migration[4.2]
4
+ def self.up
5
+ QC::Setup.update_to_4_0_0
6
+ end
7
+
8
+ def self.down
9
+ QC::Setup.downgrade_from_4_0_0
10
+ end
11
+ end
@@ -0,0 +1,86 @@
1
+ # frozen_string_literal: true
2
+
3
+ module QC
4
+ module Config
5
+ # You can use the APP_NAME to query for
6
+ # postgres related process information in the
7
+ # pg_stat_activity table.
8
+ def app_name
9
+ @app_name ||= ENV["QC_APP_NAME"] || "queue_classic"
10
+ end
11
+
12
+ # Number of seconds to block on the listen chanel for new jobs.
13
+ def wait_time
14
+ @wait_time ||= (ENV["QC_LISTEN_TIME"] || 5).to_i
15
+ end
16
+
17
+ # Why do you want to change the table name?
18
+ # Just deal with the default OK?
19
+ # If you do want to change this, you will
20
+ # need to update the PL/pgSQL lock_head() function.
21
+ # Come on. Don't do it.... Just stick with the default.
22
+ def table_name
23
+ @table_name ||= "queue_classic_jobs"
24
+ end
25
+
26
+ def queue
27
+ @queue = ENV["QUEUE"] || "default"
28
+ end
29
+
30
+ # The default queue used by `QC.enqueue`.
31
+ def default_queue
32
+ @default_queue ||= Queue.new(QC.queue)
33
+ end
34
+
35
+ def default_queue=(queue)
36
+ @default_queue = queue
37
+ end
38
+
39
+ # Each row in the table will have a column that
40
+ # notes the queue. You can point your workers
41
+ # at different queues.
42
+ def queues
43
+ @queues ||= (ENV["QUEUES"] && ENV["QUEUES"].split(",").map(&:strip)) || []
44
+ end
45
+
46
+ # Set this to 1 for strict FIFO.
47
+ # There is nothing special about 9....
48
+ def top_bound
49
+ @top_bound ||= (ENV["QC_TOP_BOUND"] || 9).to_i
50
+ end
51
+
52
+ # Set this variable if you wish for
53
+ # the worker to fork a UNIX process for
54
+ # each locked job. Remember to re-establish
55
+ # any database connections. See the worker
56
+ # for more details.
57
+ def fork_worker?
58
+ @fork_worker ||= (!ENV["QC_FORK_WORKER"].nil?)
59
+ end
60
+
61
+ # The worker class instantiated by QC's rake tasks.
62
+ def default_worker_class
63
+ @worker_class ||= (ENV["QC_DEFAULT_WORKER_CLASS"] && Kernel.const_get(ENV["QC_DEFAULT_WORKER_CLASS"]) ||
64
+ QC::Worker)
65
+
66
+ end
67
+
68
+ def default_worker_class=(worker_class)
69
+ @worker_class = worker_class
70
+ end
71
+
72
+ # reset memoized configuration
73
+ def reset_config
74
+ # TODO: we might want to think about storing these in a Hash.
75
+ @app_name = nil
76
+ @wait_time = nil
77
+ @table_name = nil
78
+ @queue = nil
79
+ @default_queue = nil
80
+ @queues = nil
81
+ @top_bound = nil
82
+ @fork_worker = nil
83
+ @worker_class = nil
84
+ end
85
+ end
86
+ end
@@ -1,27 +1,37 @@
1
+ # frozen_string_literal: true
2
+
1
3
  require 'uri'
2
4
  require 'pg'
3
5
 
4
6
  module QC
5
7
  class ConnAdapter
6
8
 
7
- attr_accessor :connection
8
- def initialize(c=nil)
9
- @connection = c.nil? ? establish_new : validate!(c)
9
+ def initialize(args={})
10
+ @active_record_connection_share = args[:active_record_connection_share]
11
+ @_connection = args[:connection]
10
12
  @mutex = Mutex.new
11
13
  end
12
14
 
15
+ def connection
16
+ if @active_record_connection_share && Object.const_defined?('ActiveRecord')
17
+ ActiveRecord::Base.connection.raw_connection
18
+ else
19
+ @_connection ||= establish_new
20
+ end
21
+ end
22
+
13
23
  def execute(stmt, *params)
14
24
  @mutex.synchronize do
15
25
  QC.log(:at => "exec_sql", :sql => stmt.inspect)
16
26
  begin
17
27
  params = nil if params.empty?
18
- r = @connection.exec(stmt, params)
28
+ r = connection.exec(stmt, params)
19
29
  result = []
20
30
  r.each {|t| result << t}
21
31
  result.length > 1 ? result : result.pop
22
- rescue PGError => e
32
+ rescue PG::Error => e
23
33
  QC.log(:error => e.inspect)
24
- @connection.reset
34
+ connection.reset
25
35
  raise
26
36
  end
27
37
  end
@@ -30,10 +40,10 @@ module QC
30
40
  def wait(time, *channels)
31
41
  @mutex.synchronize do
32
42
  listen_cmds = channels.map {|c| 'LISTEN "' + c.to_s + '"'}
33
- @connection.exec(listen_cmds.join(';'))
43
+ connection.exec(listen_cmds.join(';'))
34
44
  wait_for_notify(time)
35
45
  unlisten_cmds = channels.map {|c| 'UNLISTEN "' + c.to_s + '"'}
36
- @connection.exec(unlisten_cmds.join(';'))
46
+ connection.exec(unlisten_cmds.join(';'))
37
47
  drain_notify
38
48
  end
39
49
  end
@@ -41,23 +51,30 @@ module QC
41
51
  def disconnect
42
52
  @mutex.synchronize do
43
53
  begin
44
- @connection.close
54
+ connection.close
45
55
  rescue => e
46
56
  QC.log(:at => 'disconnect', :error => e.message)
47
57
  end
48
58
  end
49
59
  end
50
60
 
61
+ def server_version
62
+ @server_version ||= begin
63
+ version = execute("SHOW server_version_num;")["server_version_num"]
64
+ version && version.to_i
65
+ end
66
+ end
67
+
51
68
  private
52
69
 
53
70
  def wait_for_notify(t)
54
71
  Array.new.tap do |msgs|
55
- @connection.wait_for_notify(t) {|event, pid, msg| msgs << msg}
72
+ connection.wait_for_notify(t) {|event, pid, msg| msgs << msg}
56
73
  end
57
74
  end
58
75
 
59
76
  def drain_notify
60
- until @connection.notifies.nil?
77
+ until connection.notifies.nil?
61
78
  QC.log(:at => "drain_notifications")
62
79
  end
63
80
  end
@@ -70,11 +87,16 @@ module QC
70
87
 
71
88
  def establish_new
72
89
  QC.log(:at => "establish_conn")
73
- conn = PGconn.connect(*normalize_db_url(db_url))
74
- if conn.status != PGconn::CONNECTION_OK
90
+ conn = PG.connect(*normalize_db_url(db_url))
91
+ if conn.status != PG::CONNECTION_OK
75
92
  QC.log(:error => conn.error)
76
93
  end
77
- conn.exec("SET application_name = '#{QC::APP_NAME}'")
94
+
95
+ if conn.server_version < 90600
96
+ raise "This version of Queue Classic does not support Postgres older than 9.6 (90600). This version is #{conn.server_version}. If you need that support, please use an older version."
97
+ end
98
+
99
+ conn.exec("SET application_name = '#{QC.app_name}'")
78
100
  conn
79
101
  end
80
102
 
@@ -85,7 +107,7 @@ module QC
85
107
  [
86
108
  host, # host or percent-encoded socket path
87
109
  url.port || 5432,
88
- nil, '', #opts, tty
110
+ nil, nil, #opts, tty
89
111
  url.path.gsub("/",""), # database name
90
112
  url.user,
91
113
  url.password
@@ -99,6 +121,5 @@ module QC
99
121
  raise(ArgumentError, "missing QC_DATABASE_URL or DATABASE_URL")
100
122
  @db_url = URI.parse(url)
101
123
  end
102
-
103
124
  end
104
125
  end