queue_classic 0.2.1 → 0.2.2
Sign up to get free protection for your applications and to get access to all the features.
- data/lib/queue_classic/api.rb +2 -1
- data/readme.markdown +80 -18
- data/test/api_test.rb +9 -0
- metadata +2 -2
data/lib/queue_classic/api.rb
CHANGED
data/readme.markdown
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
# Queue Classic
|
2
|
-
__Beta 0.2.
|
2
|
+
__Beta 0.2.2__
|
3
3
|
|
4
|
-
__Queue Classic 0.2.
|
4
|
+
__Queue Classic 0.2.2 is in Beta.__ I have been using this library with 30-50 Heroku workers and have had great results.
|
5
5
|
|
6
6
|
I am using this in production applications and plan to maintain and support this library for a long time.
|
7
7
|
|
@@ -68,12 +68,32 @@ To get access to these tasks, Add `require 'queue_classic/tasks'` to your Rakefi
|
|
68
68
|
|
69
69
|
### Enqueue
|
70
70
|
|
71
|
-
To place a job onto the queue, you should specify a class and a class method.
|
71
|
+
To place a job onto the queue, you should specify a class and a class method. There are a few ways to enqueue:
|
72
72
|
|
73
73
|
QC.enqueue('Class.method', :arg1 => 'value1', :arg2 => 'value2')
|
74
74
|
|
75
|
+
Requires:
|
76
|
+
|
77
|
+
class Class
|
78
|
+
def self.method(args)
|
79
|
+
puts args["arg1"]
|
80
|
+
end
|
81
|
+
end
|
82
|
+
|
83
|
+
QC.enqueue('Class.method', 'value1', 'value2')
|
84
|
+
|
85
|
+
Requires:
|
86
|
+
|
87
|
+
class Class
|
88
|
+
def self.method(arg1,arg2)
|
89
|
+
puts arg1
|
90
|
+
puts arg2
|
91
|
+
end
|
92
|
+
end
|
93
|
+
|
94
|
+
|
75
95
|
The job gets stored in the jobs table with a details field set to: `{ job: Class.method, params: {arg1: value1, arg2: value2}}` (as JSON)
|
76
|
-
|
96
|
+
Here is a more concrete example of a job implementation using a Rails ActiveRecord Model:
|
77
97
|
|
78
98
|
class Invoice < ActiveRecord::Base
|
79
99
|
def self.process(invoice_id)
|
@@ -91,32 +111,67 @@ Class can be any class and method can be anything that Class will respond to. Fo
|
|
91
111
|
|
92
112
|
### Dequeue
|
93
113
|
|
94
|
-
Traditionally, a queue's dequeue operation will remove the item from the queue. However, Queue Classic will not delete the item from the queue,
|
95
|
-
and then the worker will delete the job once it has finished working it. Queue Classic's greatest strength is it's ability to safely lock jobs. Unlike other
|
96
|
-
database backed queing libraries, Classic
|
114
|
+
Traditionally, a queue's dequeue operation will remove the item from the queue. However, Queue Classic will not delete the item from the queue right away; instead, the workers will lock
|
115
|
+
the job and then the worker will delete the job once it has finished working it. Queue Classic's greatest strength is it's ability to safely lock jobs. Unlike other
|
116
|
+
database backed queing libraries, Queue Classic uses the database time to lock. This allows you to be more relaxed about the time synchronization amongst your worker machines.
|
97
117
|
|
98
|
-
|
99
|
-
|
100
|
-
|
118
|
+
Queue Classic takes advantage of Postgres' PUB/SUB featuers to dequeue a job. Basically there is a channel in which the workers LISTEN. When a new job is added to the queue, the queue sends NOTIFY
|
119
|
+
messages on the channel. Once a NOTIFY is sent, each worker races to acquire a lock on a job. A job is awareded to the victor while the rest go back to wait for another job. This eliminates
|
120
|
+
the need to Sleep & Select.
|
101
121
|
|
102
122
|
### The Worker
|
103
123
|
|
104
124
|
The worker calls dequeue and then calls the enqueued method with the supplied arguments. Once the method terminates, the job is deleted from the queue. In the case that your method
|
105
|
-
does not terminate, or the worker unexpectingly dies, Queue Classic will do following:
|
106
|
-
|
125
|
+
does not terminate, or the worker unexpectingly dies, Queue Classic will do following:
|
126
|
+
|
127
|
+
* Rescue the Exception %
|
128
|
+
* Call handle_failure(job,exception)
|
129
|
+
* Delete the job
|
130
|
+
|
131
|
+
% - To my knowledge, the only thing that can usurp ensure is a segfault.
|
132
|
+
|
133
|
+
By default, handle_failure will puts the job and the exception. This is not very good and you should override this method. It is simple to do so.
|
134
|
+
If you are using Queue Classic with Rails, You should:
|
135
|
+
|
136
|
+
1. Remove require 'queue_classic/tasks' from Rakefile
|
137
|
+
2. Create new file in lib/tasks. Call it queue_classic.rb (name is arbitrary)
|
138
|
+
3. Insert something like the following:
|
139
|
+
|
140
|
+
require 'queue_classic'
|
107
141
|
|
108
142
|
class MyWorker < QC::Worker
|
109
143
|
def handle_failure(job,exception)
|
144
|
+
# You can do many things inside of this method. Here are a few examples:
|
145
|
+
|
146
|
+
# Log to Exceptional
|
147
|
+
Exceptional.handle(exception, "Background Job Failed" + job.inspect)
|
148
|
+
|
149
|
+
# Log to Hoptoad
|
150
|
+
HoptoadNotifier.notify(
|
151
|
+
:error_class => "Background Job",
|
152
|
+
:error_message => "Special Error: #{e.message}",
|
153
|
+
:parameters => job.details
|
154
|
+
)
|
155
|
+
|
156
|
+
# Log to STDOUT (Heroku Logplex listens to stdout)
|
110
157
|
puts job.inspect
|
111
158
|
puts exception.inspect
|
159
|
+
puts exception.backtrace
|
160
|
+
|
161
|
+
# Retry the job
|
162
|
+
QC.enqueue(job)
|
163
|
+
|
112
164
|
end
|
113
165
|
end
|
114
166
|
|
115
|
-
|
116
|
-
|
167
|
+
namespace :jobs do
|
168
|
+
task :work => :environment do
|
169
|
+
MyWorker.new.start
|
170
|
+
end
|
171
|
+
end
|
117
172
|
|
118
173
|
## Performance
|
119
|
-
I am pleased at the performance of Queue Classic. It ran 3x faster than the
|
174
|
+
I am pleased at the performance of Queue Classic. It ran 3x faster than the DJ. (I have yet to benchmark redis backed queues)
|
120
175
|
|
121
176
|
ruby benchmark.rb
|
122
177
|
user system total real
|
@@ -126,6 +181,12 @@ Hardware: Mac Book Pro 2.8 GHz Intel Core i7. SSD. 4 GB memory.
|
|
126
181
|
|
127
182
|
Software: Ruby 1.9.2-p0, PostgreSQL 9.0.2
|
128
183
|
|
184
|
+
It is fast because:
|
185
|
+
|
186
|
+
* I wrote my own SQL
|
187
|
+
* I do not create many Ruby Objects
|
188
|
+
* I do not call very many methods
|
189
|
+
|
129
190
|
## FAQ
|
130
191
|
|
131
192
|
How is this different than DJ?
|
@@ -147,10 +208,11 @@ Why doesn't your queue retry failed jobs?
|
|
147
208
|
> I believe the Class method should handle any sort of exception. Also, I think
|
148
209
|
that the model you are working on should know about it's state. For instance, if you are
|
149
210
|
creating jobs for the emailing of newsletters; put a emailed_at column on your newsletter model
|
150
|
-
and then right before the job quits, touch the emailed_at column.
|
211
|
+
and then right before the job quits, touch the emailed_at column. That being said, you can do whatever you
|
212
|
+
want in handle_failure. I will not decide what is best for your application.
|
151
213
|
|
152
214
|
Can I use this library with 50 Heroku Workers?
|
153
215
|
> Yes.
|
154
216
|
|
155
|
-
|
156
|
-
> I started this project on 1/24/2011.
|
217
|
+
Is Queue Classic ready for production? Can I do it live?!?
|
218
|
+
> I started this project on 1/24/2011. I have been using this in production for some high-traffic apps at Heroku since 2/24/2011.
|
data/test/api_test.rb
CHANGED
@@ -26,6 +26,15 @@ context "QC::Api" do
|
|
26
26
|
assert_equal({"job" => "Notifier.send", "params" => []}, job.details)
|
27
27
|
end
|
28
28
|
|
29
|
+
test "enqueue takes a job and maintain params" do
|
30
|
+
h = {"id" => 1, "details" => {"job" => 'Notifier.send', "params" => ["1"]}.to_json, "locked_at" => nil}
|
31
|
+
job = QC::Job.new(h)
|
32
|
+
QC.enqueue(job)
|
33
|
+
|
34
|
+
job = QC.dequeue
|
35
|
+
assert_equal({"job" => "Notifier.send", "params" => ["1"]}, job.details)
|
36
|
+
end
|
37
|
+
|
29
38
|
|
30
39
|
end
|
31
40
|
|