queue_classic 2.0.0rc14 → 2.0.0
Sign up to get free protection for your applications and to get access to all the features.
- data/lib/queue_classic/queries.rb +1 -1
- data/lib/queue_classic/queue.rb +4 -4
- data/lib/queue_classic/tasks.rb +3 -1
- data/readme.md +43 -28
- data/sql/drop_ddl.sql +0 -1
- data/test/queue_test.rb +12 -0
- metadata +28 -18
data/lib/queue_classic/queue.rb
CHANGED
@@ -20,12 +20,12 @@ module QC
|
|
20
20
|
Queries.delete(id)
|
21
21
|
end
|
22
22
|
|
23
|
-
def delete_all
|
24
|
-
Queries.delete_all(
|
23
|
+
def delete_all
|
24
|
+
Queries.delete_all(@name)
|
25
25
|
end
|
26
26
|
|
27
|
-
def count
|
28
|
-
Queries.count(
|
27
|
+
def count
|
28
|
+
Queries.count(@name)
|
29
29
|
end
|
30
30
|
|
31
31
|
end
|
data/lib/queue_classic/tasks.rb
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
task :environment
|
2
|
+
|
1
3
|
namespace :jobs do
|
2
4
|
desc "Alias for qc:work"
|
3
5
|
task :work => "qc:work"
|
@@ -11,7 +13,7 @@ namespace :qc do
|
|
11
13
|
|
12
14
|
desc "Returns the number of jobs in the (default or QUEUE) queue"
|
13
15
|
task :count => :environment do
|
14
|
-
QC::Worker.new.queue.count
|
16
|
+
puts QC::Worker.new.queue.count
|
15
17
|
end
|
16
18
|
|
17
19
|
desc "Setup queue_classic tables and funtions in database"
|
data/readme.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
1
|
# queue_classic
|
2
2
|
|
3
|
-
v2.0.
|
3
|
+
v2.0.0
|
4
4
|
|
5
|
-
queue_classic
|
6
|
-
|
7
|
-
|
5
|
+
queue_classic provides PostgreSQL-backed queueing focused on concurrent job
|
6
|
+
locking and minimizing database load while providing a simple, intuitive user
|
7
|
+
experience.
|
8
8
|
|
9
9
|
queue_classic features:
|
10
10
|
|
@@ -19,24 +19,24 @@ queue_classic features:
|
|
19
19
|
|
20
20
|
## Proven
|
21
21
|
|
22
|
-
queue_classic was designed out of necessity
|
23
|
-
|
24
|
-
of
|
25
|
-
a small API and very few features.
|
26
|
-
|
27
|
-
|
28
|
-
|
22
|
+
queue_classic was designed out of the necessity for a fast, reliable, low
|
23
|
+
maintenance message queue. It was built upon PostgreSQL to avoid the necessity
|
24
|
+
of adding redis or 0MQ services to my applications. It was designed to be
|
25
|
+
simple, with a small API and very few features. For a simple mechanism to
|
26
|
+
distribute jobs to worker processes, especially if you are already running
|
27
|
+
PostgreSQL, queue_classic is exactly what you should be using. If you need
|
28
|
+
more advanced queueing features, you should investigate 0MQ, rabbitmq, or redis.
|
29
29
|
|
30
30
|
### Heroku Postgres
|
31
31
|
|
32
32
|
The Heroku Postgres team uses queue_classic to monitor the health of
|
33
|
-
customer databases
|
33
|
+
customer databases, processng 200 jobs per second using a [fugu](https://postgres.heroku.com/pricing)
|
34
34
|
database. They chose queue_classic because of its simplicity and reliability.
|
35
35
|
|
36
36
|
### Cloudapp
|
37
37
|
|
38
|
-
Larry uses queue_classic to deliver cloudapp's push notifications and
|
39
|
-
|
38
|
+
Larry uses queue_classic to deliver cloudapp's push notifications and collect
|
39
|
+
file meta-data from S3, processing nearly 14 jobs per second.
|
40
40
|
|
41
41
|
```
|
42
42
|
I haven't even touched QC since setting it up.
|
@@ -48,8 +48,8 @@ The best queue is the one you don't have to hand hold.
|
|
48
48
|
## Setup
|
49
49
|
|
50
50
|
In addition to installing the rubygem, you will need to prepare your database.
|
51
|
-
Database
|
52
|
-
You can issue the database
|
51
|
+
Database preparation includes creating a table and loading PL/pgSQL functions.
|
52
|
+
You can issue the database preparation commands using **PSQL(1)** or place them in a
|
53
53
|
database migration.
|
54
54
|
|
55
55
|
### Quick Start
|
@@ -69,7 +69,7 @@ $ ruby -r queue_classic -e "QC::Worker.new.work"
|
|
69
69
|
|
70
70
|
```ruby
|
71
71
|
source :rubygems
|
72
|
-
gem "queue_classic", "2.0.
|
72
|
+
gem "queue_classic", "2.0.0"
|
73
73
|
```
|
74
74
|
|
75
75
|
**Rakefile**
|
@@ -86,6 +86,10 @@ require "queue_classic/tasks"
|
|
86
86
|
ENV["DATABASE_URL"] = "postgres://username:password@localhost/database_name"
|
87
87
|
```
|
88
88
|
|
89
|
+
queue_classic requires a database table and a PL/pgSQL function to be loaded
|
90
|
+
into your database. You can load the table and the function by running a migration
|
91
|
+
or using a rake task.
|
92
|
+
|
89
93
|
**db/migrations/add_queue_classic.rb**
|
90
94
|
|
91
95
|
```ruby
|
@@ -104,6 +108,17 @@ class AddQueueClassic < ActiveRecord::Migration
|
|
104
108
|
end
|
105
109
|
```
|
106
110
|
|
111
|
+
**Rake Task**
|
112
|
+
|
113
|
+
```bash
|
114
|
+
# Creating the table and functions
|
115
|
+
$ bundle exec rake qc:create
|
116
|
+
|
117
|
+
# Dropping the table and functions
|
118
|
+
$ bundle exec rake qc:drop
|
119
|
+
```
|
120
|
+
|
121
|
+
|
107
122
|
### Sequel Setup
|
108
123
|
|
109
124
|
**db/migrations/1_add_queue_classic.rb**
|
@@ -179,7 +194,7 @@ read up on [OkJson](https://github.com/kr/okjson)
|
|
179
194
|
|
180
195
|
The table containing the jobs has a column named *q_name*. This column
|
181
196
|
is the abstraction queue_classic uses to represent multiple queues. This allows
|
182
|
-
the programmer to place triggers and
|
197
|
+
the programmer to place triggers and indexes on distinct queues.
|
183
198
|
|
184
199
|
```ruby
|
185
200
|
# attach to the priority_queue. this will insert
|
@@ -203,7 +218,7 @@ p_queue.enqueue("Kernel.puts", ["hello", "world"])
|
|
203
218
|
```
|
204
219
|
|
205
220
|
This code example shows how to produce jobs into a custom queue,
|
206
|
-
to consume jobs from the
|
221
|
+
to consume jobs from the custom queue be sure and set the `$QUEUE`
|
207
222
|
var to the q_name in the worker's UNIX environment.
|
208
223
|
|
209
224
|
### Consumer
|
@@ -263,7 +278,7 @@ class MyWorker < QC::Worker
|
|
263
278
|
|
264
279
|
# retry the job
|
265
280
|
def handle_failure(job, exception)
|
266
|
-
@queue.
|
281
|
+
@queue.enqueue(job[:method], job[:args])
|
267
282
|
end
|
268
283
|
|
269
284
|
# the forked proc needs a new db connection
|
@@ -358,9 +373,9 @@ $ export QC_LISTENING_WORKER='true'
|
|
358
373
|
|
359
374
|
##### Failure
|
360
375
|
|
361
|
-
I bet your worker will encounter a job that raises an exception.
|
376
|
+
I bet your worker will encounter a job that raises an exception. queue_classic
|
362
377
|
thinks that you should know about this exception by means of you established
|
363
|
-
exception tracker. (i.e. Hoptoad, Exceptional) To that end,
|
378
|
+
exception tracker. (i.e. Hoptoad, Exceptional) To that end, queue_classic offers
|
364
379
|
a method that you can override. This method will be passed 2 arguments: the
|
365
380
|
exception instance and the job. Here are a few examples of things you might want
|
366
381
|
to do inside `handle_failure()`.
|
@@ -382,19 +397,19 @@ lib=queue_classic level=info action=insert_job elapsed=16
|
|
382
397
|
|
383
398
|
Author: [@em_csquared](https://twitter.com/#!/em_csquared)
|
384
399
|
|
385
|
-
I was
|
400
|
+
I was testing some code that started out handling some work in a web request and
|
386
401
|
wanted to move that work over to a queue. After completing a red-green-refactor
|
387
402
|
I did not want my tests to have to worry about workers or even hit the database.
|
388
403
|
|
389
|
-
Turns out its easy to get
|
404
|
+
Turns out its easy to get queue_classic to just work in a synchronous way with:
|
390
405
|
|
391
406
|
```ruby
|
392
407
|
def QC.enqueue(function_call, *args)
|
393
|
-
eval("#{function_call}
|
408
|
+
eval("#{function_call} *#{args.inspect}")
|
394
409
|
end
|
395
410
|
```
|
396
411
|
|
397
|
-
Now you can test
|
412
|
+
Now you can test queue_classic as if it was calling your method directly!
|
398
413
|
|
399
414
|
|
400
415
|
### Dispatching new jobs to workers without new code
|
@@ -409,7 +424,7 @@ these objects. queue_classic to the rescue! (no pun intended)
|
|
409
424
|
|
410
425
|
The API of queue_classic enables you to quickly dispatch jobs to workers. In my
|
411
426
|
case I wanted to call `Invoice.destroy(id)` a few thousand times. I fired up a
|
412
|
-
|
427
|
+
Heroku console session and executed this line:
|
413
428
|
|
414
429
|
```ruby
|
415
430
|
Invoice.find(:all, :select => "id", :conditions => "some condition").map {|i| QC.enqueue("Invoice.destroy", i.id) }
|
@@ -490,7 +505,7 @@ SalesSummaryGenerator class.
|
|
490
505
|
|
491
506
|
I found this abstraction quite powerful and easy to understand. Like
|
492
507
|
queue_classic, the clockwork gem is simple to understand and has 0 dependencies.
|
493
|
-
In production, I create a
|
508
|
+
In production, I create a Heroku process type called clock. This is typically
|
494
509
|
what my Procfile looks like:
|
495
510
|
|
496
511
|
```
|
data/sql/drop_ddl.sql
CHANGED
data/test/queue_test.rb
CHANGED
@@ -36,6 +36,18 @@ class QueueTest < QCTest
|
|
36
36
|
assert_equal(0, QC.count)
|
37
37
|
end
|
38
38
|
|
39
|
+
def test_delete_all_by_queue_name
|
40
|
+
p_queue = QC::Queue.new("priority_queue")
|
41
|
+
s_queue = QC::Queue.new("secondary_queue")
|
42
|
+
p_queue.enqueue("Klass.method")
|
43
|
+
s_queue.enqueue("Klass.method")
|
44
|
+
assert_equal(1, p_queue.count)
|
45
|
+
assert_equal(1, s_queue.count)
|
46
|
+
p_queue.delete_all
|
47
|
+
assert_equal(0, p_queue.count)
|
48
|
+
assert_equal(1, s_queue.count)
|
49
|
+
end
|
50
|
+
|
39
51
|
def test_queue_instance
|
40
52
|
queue = QC::Queue.new("queue_classic_jobs", false)
|
41
53
|
queue.enqueue("Klass.method")
|
metadata
CHANGED
@@ -1,19 +1,19 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: queue_classic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.0.
|
5
|
-
prerelease:
|
4
|
+
version: 2.0.0
|
5
|
+
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
|
-
- Ryan Smith (ace hacker)
|
8
|
+
- Ryan Smith (♠ ace hacker)
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-02-29 00:00:00.
|
12
|
+
date: 2012-02-29 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: pg
|
16
|
-
requirement:
|
16
|
+
requirement: !ruby/object:Gem::Requirement
|
17
17
|
none: false
|
18
18
|
requirements:
|
19
19
|
- - ~>
|
@@ -21,10 +21,15 @@ dependencies:
|
|
21
21
|
version: 0.13.2
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
|
-
version_requirements:
|
24
|
+
version_requirements: !ruby/object:Gem::Requirement
|
25
|
+
none: false
|
26
|
+
requirements:
|
27
|
+
- - ~>
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: 0.13.2
|
25
30
|
- !ruby/object:Gem::Dependency
|
26
31
|
name: scrolls
|
27
|
-
requirement:
|
32
|
+
requirement: !ruby/object:Gem::Requirement
|
28
33
|
none: false
|
29
34
|
requirements:
|
30
35
|
- - ~>
|
@@ -32,7 +37,12 @@ dependencies:
|
|
32
37
|
version: 0.0.8
|
33
38
|
type: :runtime
|
34
39
|
prerelease: false
|
35
|
-
version_requirements:
|
40
|
+
version_requirements: !ruby/object:Gem::Requirement
|
41
|
+
none: false
|
42
|
+
requirements:
|
43
|
+
- - ~>
|
44
|
+
- !ruby/object:Gem::Version
|
45
|
+
version: 0.0.8
|
36
46
|
description: queue_classic is a queueing library for Ruby apps. (Rails, Sinatra, Etc...)
|
37
47
|
queue_classic features asynchronous job polling, database maintained locks and no
|
38
48
|
ridiculous dependencies. As a matter of fact, queue_classic only requires pg.
|
@@ -42,20 +52,20 @@ extensions: []
|
|
42
52
|
extra_rdoc_files: []
|
43
53
|
files:
|
44
54
|
- readme.md
|
45
|
-
- sql/ddl.sql
|
46
55
|
- sql/create_table.sql
|
56
|
+
- sql/ddl.sql
|
47
57
|
- sql/drop_ddl.sql
|
48
58
|
- lib/queue_classic/conn.rb
|
49
59
|
- lib/queue_classic/okjson.rb
|
50
|
-
- lib/queue_classic/
|
60
|
+
- lib/queue_classic/queries.rb
|
51
61
|
- lib/queue_classic/queue.rb
|
52
|
-
- lib/queue_classic/tasks.rb
|
53
62
|
- lib/queue_classic/setup.rb
|
54
|
-
- lib/queue_classic/
|
63
|
+
- lib/queue_classic/tasks.rb
|
64
|
+
- lib/queue_classic/worker.rb
|
55
65
|
- lib/queue_classic.rb
|
56
|
-
- test/worker_test.rb
|
57
|
-
- test/queue_test.rb
|
58
66
|
- test/helper.rb
|
67
|
+
- test/queue_test.rb
|
68
|
+
- test/worker_test.rb
|
59
69
|
homepage: http://github.com/ryandotsmith/queue_classic
|
60
70
|
licenses: []
|
61
71
|
post_install_message:
|
@@ -71,15 +81,15 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
71
81
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
72
82
|
none: false
|
73
83
|
requirements:
|
74
|
-
- - ! '
|
84
|
+
- - ! '>='
|
75
85
|
- !ruby/object:Gem::Version
|
76
|
-
version:
|
86
|
+
version: '0'
|
77
87
|
requirements: []
|
78
88
|
rubyforge_project:
|
79
|
-
rubygems_version: 1.8.
|
89
|
+
rubygems_version: 1.8.23
|
80
90
|
signing_key:
|
81
91
|
specification_version: 3
|
82
92
|
summary: postgres backed queue
|
83
93
|
test_files:
|
84
|
-
- test/worker_test.rb
|
85
94
|
- test/queue_test.rb
|
95
|
+
- test/worker_test.rb
|