queue_classic 2.0.0rc14 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/lib/queue_classic/queries.rb +1 -1
- data/lib/queue_classic/queue.rb +4 -4
- data/lib/queue_classic/tasks.rb +3 -1
- data/readme.md +43 -28
- data/sql/drop_ddl.sql +0 -1
- data/test/queue_test.rb +12 -0
- metadata +28 -18
data/lib/queue_classic/queue.rb
CHANGED
@@ -20,12 +20,12 @@ module QC
|
|
20
20
|
Queries.delete(id)
|
21
21
|
end
|
22
22
|
|
23
|
-
def delete_all
|
24
|
-
Queries.delete_all(
|
23
|
+
def delete_all
|
24
|
+
Queries.delete_all(@name)
|
25
25
|
end
|
26
26
|
|
27
|
-
def count
|
28
|
-
Queries.count(
|
27
|
+
def count
|
28
|
+
Queries.count(@name)
|
29
29
|
end
|
30
30
|
|
31
31
|
end
|
data/lib/queue_classic/tasks.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
task :environment
|
2
|
+
|
1
3
|
namespace :jobs do
|
2
4
|
desc "Alias for qc:work"
|
3
5
|
task :work => "qc:work"
|
@@ -11,7 +13,7 @@ namespace :qc do
|
|
11
13
|
|
12
14
|
desc "Returns the number of jobs in the (default or QUEUE) queue"
|
13
15
|
task :count => :environment do
|
14
|
-
QC::Worker.new.queue.count
|
16
|
+
puts QC::Worker.new.queue.count
|
15
17
|
end
|
16
18
|
|
17
19
|
desc "Setup queue_classic tables and funtions in database"
|
data/readme.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
1
|
# queue_classic
|
2
2
|
|
3
|
-
v2.0.
|
3
|
+
v2.0.0
|
4
4
|
|
5
|
-
queue_classic
|
6
|
-
|
7
|
-
|
5
|
+
queue_classic provides PostgreSQL-backed queueing focused on concurrent job
|
6
|
+
locking and minimizing database load while providing a simple, intuitive user
|
7
|
+
experience.
|
8
8
|
|
9
9
|
queue_classic features:
|
10
10
|
|
@@ -19,24 +19,24 @@ queue_classic features:
|
|
19
19
|
|
20
20
|
## Proven
|
21
21
|
|
22
|
-
queue_classic was designed out of necessity
|
23
|
-
|
24
|
-
of
|
25
|
-
a small API and very few features.
|
26
|
-
|
27
|
-
|
28
|
-
|
22
|
+
queue_classic was designed out of the necessity for a fast, reliable, low
|
23
|
+
maintenance message queue. It was built upon PostgreSQL to avoid the necessity
|
24
|
+
of adding redis or 0MQ services to my applications. It was designed to be
|
25
|
+
simple, with a small API and very few features. For a simple mechanism to
|
26
|
+
distribute jobs to worker processes, especially if you are already running
|
27
|
+
PostgreSQL, queue_classic is exactly what you should be using. If you need
|
28
|
+
more advanced queueing features, you should investigate 0MQ, rabbitmq, or redis.
|
29
29
|
|
30
30
|
### Heroku Postgres
|
31
31
|
|
32
32
|
The Heroku Postgres team uses queue_classic to monitor the health of
|
33
|
-
customer databases
|
33
|
+
customer databases, processng 200 jobs per second using a [fugu](https://postgres.heroku.com/pricing)
|
34
34
|
database. They chose queue_classic because of its simplicity and reliability.
|
35
35
|
|
36
36
|
### Cloudapp
|
37
37
|
|
38
|
-
Larry uses queue_classic to deliver cloudapp's push notifications and
|
39
|
-
|
38
|
+
Larry uses queue_classic to deliver cloudapp's push notifications and collect
|
39
|
+
file meta-data from S3, processing nearly 14 jobs per second.
|
40
40
|
|
41
41
|
```
|
42
42
|
I haven't even touched QC since setting it up.
|
@@ -48,8 +48,8 @@ The best queue is the one you don't have to hand hold.
|
|
48
48
|
## Setup
|
49
49
|
|
50
50
|
In addition to installing the rubygem, you will need to prepare your database.
|
51
|
-
Database
|
52
|
-
You can issue the database
|
51
|
+
Database preparation includes creating a table and loading PL/pgSQL functions.
|
52
|
+
You can issue the database preparation commands using **PSQL(1)** or place them in a
|
53
53
|
database migration.
|
54
54
|
|
55
55
|
### Quick Start
|
@@ -69,7 +69,7 @@ $ ruby -r queue_classic -e "QC::Worker.new.work"
|
|
69
69
|
|
70
70
|
```ruby
|
71
71
|
source :rubygems
|
72
|
-
gem "queue_classic", "2.0.
|
72
|
+
gem "queue_classic", "2.0.0"
|
73
73
|
```
|
74
74
|
|
75
75
|
**Rakefile**
|
@@ -86,6 +86,10 @@ require "queue_classic/tasks"
|
|
86
86
|
ENV["DATABASE_URL"] = "postgres://username:password@localhost/database_name"
|
87
87
|
```
|
88
88
|
|
89
|
+
queue_classic requires a database table and a PL/pgSQL function to be loaded
|
90
|
+
into your database. You can load the table and the function by running a migration
|
91
|
+
or using a rake task.
|
92
|
+
|
89
93
|
**db/migrations/add_queue_classic.rb**
|
90
94
|
|
91
95
|
```ruby
|
@@ -104,6 +108,17 @@ class AddQueueClassic < ActiveRecord::Migration
|
|
104
108
|
end
|
105
109
|
```
|
106
110
|
|
111
|
+
**Rake Task**
|
112
|
+
|
113
|
+
```bash
|
114
|
+
# Creating the table and functions
|
115
|
+
$ bundle exec rake qc:create
|
116
|
+
|
117
|
+
# Dropping the table and functions
|
118
|
+
$ bundle exec rake qc:drop
|
119
|
+
```
|
120
|
+
|
121
|
+
|
107
122
|
### Sequel Setup
|
108
123
|
|
109
124
|
**db/migrations/1_add_queue_classic.rb**
|
@@ -179,7 +194,7 @@ read up on [OkJson](https://github.com/kr/okjson)
|
|
179
194
|
|
180
195
|
The table containing the jobs has a column named *q_name*. This column
|
181
196
|
is the abstraction queue_classic uses to represent multiple queues. This allows
|
182
|
-
the programmer to place triggers and
|
197
|
+
the programmer to place triggers and indexes on distinct queues.
|
183
198
|
|
184
199
|
```ruby
|
185
200
|
# attach to the priority_queue. this will insert
|
@@ -203,7 +218,7 @@ p_queue.enqueue("Kernel.puts", ["hello", "world"])
|
|
203
218
|
```
|
204
219
|
|
205
220
|
This code example shows how to produce jobs into a custom queue,
|
206
|
-
to consume jobs from the
|
221
|
+
to consume jobs from the custom queue be sure and set the `$QUEUE`
|
207
222
|
var to the q_name in the worker's UNIX environment.
|
208
223
|
|
209
224
|
### Consumer
|
@@ -263,7 +278,7 @@ class MyWorker < QC::Worker
|
|
263
278
|
|
264
279
|
# retry the job
|
265
280
|
def handle_failure(job, exception)
|
266
|
-
@queue.
|
281
|
+
@queue.enqueue(job[:method], job[:args])
|
267
282
|
end
|
268
283
|
|
269
284
|
# the forked proc needs a new db connection
|
@@ -358,9 +373,9 @@ $ export QC_LISTENING_WORKER='true'
|
|
358
373
|
|
359
374
|
##### Failure
|
360
375
|
|
361
|
-
I bet your worker will encounter a job that raises an exception.
|
376
|
+
I bet your worker will encounter a job that raises an exception. queue_classic
|
362
377
|
thinks that you should know about this exception by means of you established
|
363
|
-
exception tracker. (i.e. Hoptoad, Exceptional) To that end,
|
378
|
+
exception tracker. (i.e. Hoptoad, Exceptional) To that end, queue_classic offers
|
364
379
|
a method that you can override. This method will be passed 2 arguments: the
|
365
380
|
exception instance and the job. Here are a few examples of things you might want
|
366
381
|
to do inside `handle_failure()`.
|
@@ -382,19 +397,19 @@ lib=queue_classic level=info action=insert_job elapsed=16
|
|
382
397
|
|
383
398
|
Author: [@em_csquared](https://twitter.com/#!/em_csquared)
|
384
399
|
|
385
|
-
I was
|
400
|
+
I was testing some code that started out handling some work in a web request and
|
386
401
|
wanted to move that work over to a queue. After completing a red-green-refactor
|
387
402
|
I did not want my tests to have to worry about workers or even hit the database.
|
388
403
|
|
389
|
-
Turns out its easy to get
|
404
|
+
Turns out its easy to get queue_classic to just work in a synchronous way with:
|
390
405
|
|
391
406
|
```ruby
|
392
407
|
def QC.enqueue(function_call, *args)
|
393
|
-
eval("#{function_call}
|
408
|
+
eval("#{function_call} *#{args.inspect}")
|
394
409
|
end
|
395
410
|
```
|
396
411
|
|
397
|
-
Now you can test
|
412
|
+
Now you can test queue_classic as if it was calling your method directly!
|
398
413
|
|
399
414
|
|
400
415
|
### Dispatching new jobs to workers without new code
|
@@ -409,7 +424,7 @@ these objects. queue_classic to the rescue! (no pun intended)
|
|
409
424
|
|
410
425
|
The API of queue_classic enables you to quickly dispatch jobs to workers. In my
|
411
426
|
case I wanted to call `Invoice.destroy(id)` a few thousand times. I fired up a
|
412
|
-
|
427
|
+
Heroku console session and executed this line:
|
413
428
|
|
414
429
|
```ruby
|
415
430
|
Invoice.find(:all, :select => "id", :conditions => "some condition").map {|i| QC.enqueue("Invoice.destroy", i.id) }
|
@@ -490,7 +505,7 @@ SalesSummaryGenerator class.
|
|
490
505
|
|
491
506
|
I found this abstraction quite powerful and easy to understand. Like
|
492
507
|
queue_classic, the clockwork gem is simple to understand and has 0 dependencies.
|
493
|
-
In production, I create a
|
508
|
+
In production, I create a Heroku process type called clock. This is typically
|
494
509
|
what my Procfile looks like:
|
495
510
|
|
496
511
|
```
|
data/sql/drop_ddl.sql
CHANGED
data/test/queue_test.rb
CHANGED
@@ -36,6 +36,18 @@ class QueueTest < QCTest
|
|
36
36
|
assert_equal(0, QC.count)
|
37
37
|
end
|
38
38
|
|
39
|
+
def test_delete_all_by_queue_name
|
40
|
+
p_queue = QC::Queue.new("priority_queue")
|
41
|
+
s_queue = QC::Queue.new("secondary_queue")
|
42
|
+
p_queue.enqueue("Klass.method")
|
43
|
+
s_queue.enqueue("Klass.method")
|
44
|
+
assert_equal(1, p_queue.count)
|
45
|
+
assert_equal(1, s_queue.count)
|
46
|
+
p_queue.delete_all
|
47
|
+
assert_equal(0, p_queue.count)
|
48
|
+
assert_equal(1, s_queue.count)
|
49
|
+
end
|
50
|
+
|
39
51
|
def test_queue_instance
|
40
52
|
queue = QC::Queue.new("queue_classic_jobs", false)
|
41
53
|
queue.enqueue("Klass.method")
|
metadata
CHANGED
@@ -1,19 +1,19 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: queue_classic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.0.
|
5
|
-
prerelease:
|
4
|
+
version: 2.0.0
|
5
|
+
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
|
-
- Ryan Smith (ace hacker)
|
8
|
+
- Ryan Smith (♠ ace hacker)
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-02-29 00:00:00.
|
12
|
+
date: 2012-02-29 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: pg
|
16
|
-
requirement:
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
17
|
none: false
|
18
18
|
requirements:
|
19
19
|
- - ~>
|
@@ -21,10 +21,15 @@ dependencies:
|
|
21
21
|
version: 0.13.2
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
|
-
version_requirements:
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.13.2
|
25
30
|
- !ruby/object:Gem::Dependency
|
26
31
|
name: scrolls
|
27
|
-
requirement:
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
28
33
|
none: false
|
29
34
|
requirements:
|
30
35
|
- - ~>
|
@@ -32,7 +37,12 @@ dependencies:
|
|
32
37
|
version: 0.0.8
|
33
38
|
type: :runtime
|
34
39
|
prerelease: false
|
35
|
-
version_requirements:
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ~>
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: 0.0.8
|
36
46
|
description: queue_classic is a queueing library for Ruby apps. (Rails, Sinatra, Etc...)
|
37
47
|
queue_classic features asynchronous job polling, database maintained locks and no
|
38
48
|
ridiculous dependencies. As a matter of fact, queue_classic only requires pg.
|
@@ -42,20 +52,20 @@ extensions: []
|
|
42
52
|
extra_rdoc_files: []
|
43
53
|
files:
|
44
54
|
- readme.md
|
45
|
-
- sql/ddl.sql
|
46
55
|
- sql/create_table.sql
|
56
|
+
- sql/ddl.sql
|
47
57
|
- sql/drop_ddl.sql
|
48
58
|
- lib/queue_classic/conn.rb
|
49
59
|
- lib/queue_classic/okjson.rb
|
50
|
-
- lib/queue_classic/
|
60
|
+
- lib/queue_classic/queries.rb
|
51
61
|
- lib/queue_classic/queue.rb
|
52
|
-
- lib/queue_classic/tasks.rb
|
53
62
|
- lib/queue_classic/setup.rb
|
54
|
-
- lib/queue_classic/
|
63
|
+
- lib/queue_classic/tasks.rb
|
64
|
+
- lib/queue_classic/worker.rb
|
55
65
|
- lib/queue_classic.rb
|
56
|
-
- test/worker_test.rb
|
57
|
-
- test/queue_test.rb
|
58
66
|
- test/helper.rb
|
67
|
+
- test/queue_test.rb
|
68
|
+
- test/worker_test.rb
|
59
69
|
homepage: http://github.com/ryandotsmith/queue_classic
|
60
70
|
licenses: []
|
61
71
|
post_install_message:
|
@@ -71,15 +81,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
71
81
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
72
82
|
none: false
|
73
83
|
requirements:
|
74
|
-
- - ! '
|
84
|
+
- - ! '>='
|
75
85
|
- !ruby/object:Gem::Version
|
76
|
-
version:
|
86
|
+
version: '0'
|
77
87
|
requirements: []
|
78
88
|
rubyforge_project:
|
79
|
-
rubygems_version: 1.8.
|
89
|
+
rubygems_version: 1.8.23
|
80
90
|
signing_key:
|
81
91
|
specification_version: 3
|
82
92
|
summary: postgres backed queue
|
83
93
|
test_files:
|
84
|
-
- test/worker_test.rb
|
85
94
|
- test/queue_test.rb
|
95
|
+
- test/worker_test.rb
|