q 0.0.0 → 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: e3cdf838ecd7a6e78ca01649ef8ddb81bfc2ba0e
4
+ data.tar.gz: 1127ec257eae0466e467399aad375be089beef4c
5
+ SHA512:
6
+ metadata.gz: cf757de052b53a97d4d49b7087cebaa525dbef6def2ab5d2ec61f74e5700e3422e78a9e556c5992158a224cc4d2268ec80eab437d884a3050d092290ae7c44b3
7
+ data.tar.gz: 3de8a51266102381119c43a7d03d4686d0136e943c3d16c7947438b0695383c00655141e7f53e86f351f1d917335a942a577d5e47855cf1a45dd5e4e88687dcf
data/.gitignore CHANGED
@@ -1,6 +1,6 @@
1
+ .DS_Store
2
+ test/fixtures/repos/*
1
3
  *.gem
2
- .bundle
4
+
3
5
  Gemfile.lock
4
- pkg/*
5
- *.db
6
- *.rbc
6
+ test/db/
@@ -0,0 +1,11 @@
1
+ ## 0.0.3
2
+
3
+ - Bugfix: fix queue name
4
+
5
+ ## 0.0.2
6
+
7
+ - Allow early returns from blocks.
8
+
9
+ ## 0.0.1
10
+
11
+ - Initial commit
data/Gemfile CHANGED
@@ -1,4 +1,3 @@
1
- source "http://rubygems.org"
1
+ source "https://rubygems.org"
2
2
 
3
- # Specify your gem's dependencies in q.gemspec
4
- gemspec
3
+ gemspec
@@ -0,0 +1,444 @@
1
+ # Queue Authors
2
+
3
+ This document explains how to make your queuing library compatible with the `Q` interface.
4
+
5
+ You can see existing implementations in `lib/q/methods` for reference implementations.
6
+
7
+ ## Background
8
+
9
+ The primary focus of this project is usability for the end user. Ideally they should know as little as possible about a queueing implementation to be able to push a job to the background. To fully manage a queuing library we need to be able to do 4 things:
10
+
11
+ - Create background tasks
12
+ - Enqueue background tasks
13
+ - Run background tasks
14
+ - Configure the background library
15
+
16
+ To integrate with `Q` you need to add these abilities programatically. While `Q::Methods::Base` does a lot of heavy lifting (as you'll see) you still need to do a fair amount of meta programming to make all this possible.
17
+
18
+ For demonstration sake we'll use a fictitious background queuing library called `FooQueue`. It will be very similar to Resque 1.x in implementation.
19
+
20
+ ## How to Include Your Module with Q
21
+
22
+ There are default implementations for several queuing libraries included with `Q` if you own one of them, we invite you to add `Q` as a gem requirement, and then to over-write over-write
23
+
24
+ ## Basics
25
+
26
+ You will need to create a module that is included in other classes/modules. Create a module in your gem `lib/q/methods/foo_queue.rb`. Inside of that file you'll need to define your module and include the `Q::Methods::Base` module:
27
+
28
+ ```ruby
29
+ module Q::Methods::FooQueue
30
+ include Q::Methods::Base
31
+ end
32
+ ```
33
+
34
+ Of course this will mean that you have `require 'q'` somewhere previously in your code. Everything we need to integrate with `Q` will go in this module.
35
+
36
+ To work you'll need the following constants defined in your code:
37
+
38
+ - QueueMethod
39
+ - QueueBuild
40
+ - QueueTask
41
+ - QueueConfig
42
+
43
+ You can add them as classes:
44
+
45
+ ```ruby
46
+ module Q::Methods::FooQueue
47
+ include Q::Methods::Base
48
+
49
+ class QueueBuild
50
+ end
51
+ class QueueMethod
52
+ end
53
+ class QueueTask
54
+ end
55
+ class QueueConfig
56
+ end
57
+ end
58
+ ```
59
+
60
+ ## Create Background Tasks
61
+
62
+ Without the `Q` library you would need to define a `FooQueue` task as a class:
63
+
64
+ ```
65
+ class SendEmail
66
+ @queue = :send_email
67
+
68
+ def self.perform(id, state)
69
+ user = User.find(id)
70
+ issues = user.issues.where(state: state).all
71
+ UserMailer.send_issues(user: user, issues: issues).deliver
72
+ end
73
+ end
74
+ ```
75
+
76
+ You need to be able to build the exact same class with `Q` programatically. To do this you need to add a `QueueBuild` constant to your module:
77
+
78
+ ```ruby
79
+ module Q::Methods::FooQueue
80
+ include Q::Methods::Base
81
+
82
+ class QueueBuild
83
+ end
84
+ end
85
+ ```
86
+
87
+ This `QueueBuild` constant needs to respond to `call` and accept options hash and a proc with the task contents. In this case the proc it receives will look something like this:
88
+
89
+ ```ruby
90
+ Proc.new do |id, state|
91
+ user = User.find(id)
92
+ issues = user.issues.where(state: state).all
93
+ UserMailer.send_issues(user: user, issues: issues).deliver
94
+ end
95
+ ```
96
+
97
+ The options will contain:
98
+
99
+
100
+ ```ruby
101
+ options[:base] # this is the class/module that your code is being included into
102
+ options[:queue_name] # in our example this would be :send_email
103
+ options[:queue_klass_name] # in our example this would be SendEmail
104
+ ```
105
+
106
+ You can pull these out:
107
+
108
+ ```ruby
109
+ class QueueBuild
110
+ def self.call(options={}, &job)
111
+ base = options[:base]
112
+ queue_name = options[:queue_name]
113
+ queue_klass_name = options[:queue_klass_name]
114
+ end
115
+ end
116
+ ```
117
+
118
+ Now we want to build a new class that will hold our task:
119
+
120
+ ```ruby
121
+ class QueueBuild
122
+ def self.call(options={}, &job)
123
+ base = options[:base]
124
+ queue_name = options[:queue_name]
125
+ queue_klass_name = options[:queue_klass_name]
126
+
127
+ queue_klass = Class.new do
128
+ end
129
+ end
130
+ end
131
+ ```
132
+
133
+ We need a perform method and to set the `@queue` instance variable on the class, and pass in the job, we can do this by creating accessors on our class:
134
+
135
+ ```ruby
136
+ class QueueBuild
137
+ def self.call(options={}, &job)
138
+ base = options[:base]
139
+ queue_name = options[:queue_name]
140
+ queue_klass_name = options[:queue_klass_name]
141
+
142
+ queue_klass = Class.new do
143
+ def self.perform(*args)
144
+ @job.call(*args)
145
+ end
146
+
147
+ def self.job=(job)
148
+ @job = job
149
+ end
150
+
151
+ def self.queue=(queue)
152
+ @queue = queue
153
+ end
154
+ end
155
+ end
156
+ end
157
+ ```
158
+
159
+ This example will get a bit longer as we go, try to focus on the new code to keep from being overwhelmed.
160
+
161
+ Now that we can set our `@queue` and `@job` instance variables, we need to do so:
162
+
163
+ ```ruby
164
+ class QueueBuild
165
+ def self.call(options={}, &job)
166
+ base = options[:base]
167
+ queue_name = options[:queue_name]
168
+ queue_klass_name = options[:queue_klass_name]
169
+
170
+ queue_klass = Class.new do
171
+ def self.perform(*args)
172
+ @job.call(*args)
173
+ end
174
+
175
+ def self.job=(job)
176
+ @job = job
177
+ end
178
+
179
+ def self.queue=(queue)
180
+ @queue = queue
181
+ end
182
+
183
+
184
+ queue_klass.job = job
185
+ queue_klass.queue = queue_name
186
+ end
187
+ end
188
+ end
189
+ ```
190
+
191
+ We're almost done, the last thing we need is to assign this anonomous class to a constant so we can find it when we want to do the work:
192
+
193
+
194
+ ```ruby
195
+ class QueueBuild
196
+ def self.call(options={}, &job)
197
+ base = options[:base]
198
+ queue_name = options[:queue_name]
199
+ queue_klass_name = options[:queue_klass_name]
200
+
201
+ queue_klass = Class.new do
202
+ def self.perform(*args)
203
+ @job.call(*args)
204
+ end
205
+
206
+ def self.job=(job)
207
+ @job = job
208
+ end
209
+
210
+ def self.queue=(queue)
211
+ @queue = queue
212
+ end
213
+
214
+
215
+ queue_klass.job = job
216
+ queue_klass.queue = queue_name
217
+
218
+ base.const_set(queue_klass_name, queue_klass)
219
+ end
220
+ end
221
+ end
222
+ ```
223
+
224
+ Now when this code gets run by the `Q` library, this:
225
+
226
+ ```ruby
227
+ class Poro
228
+ include Q::Methods::FooQueue
229
+
230
+ queue(:send_issues) do |id, state|
231
+ user = User.find(id)
232
+ issues = user.issues.where(state: state).all
233
+ UserMailer.send_issues(user: user, issues: issues).deliver
234
+ end
235
+ end
236
+ ```
237
+
238
+ Will produce the equivalent code of this:
239
+
240
+ ```
241
+ class Poro
242
+ class SendEmail
243
+ @queue = :send_email
244
+
245
+ def self.perform(id, state)
246
+ user = User.find(id)
247
+ issues = user.issues.where(state: state).all
248
+ UserMailer.send_issues(user: user, issues: issues).deliver
249
+ end
250
+ end
251
+ end
252
+ ```
253
+
254
+ It may not look like much of a savings, but the nice this is that someone who wrote their code to use say the `Resque` backend can now switch to the `FooQueue` backend with little or no code changes.
255
+
256
+ ## Naming Convention
257
+
258
+ The name of your module is important. With `Q` you can specify the queueing library in an initializer like this:
259
+
260
+ ```ruby
261
+ Q.setup do |config|
262
+ config.queue = :resque
263
+ end
264
+ ```
265
+
266
+ Since you created a `Q::Methods::FooQueue` module a user can now configure their queue:
267
+
268
+
269
+ ```ruby
270
+ Q.setup do |config|
271
+ config.queue = :foo_queue
272
+ end
273
+ ```
274
+
275
+ If you want to be more precise you can bypass the symbol to module lookup by haing your user pass in a module directly:
276
+
277
+ ```ruby
278
+ Q.setup do |config|
279
+ config.queue = Q::Methods::FooQueue
280
+ end
281
+ ```
282
+
283
+ Once configured users can use the generic include:
284
+
285
+ ```ruby
286
+ class Poro
287
+ include Q::Methods
288
+ end
289
+ ```
290
+
291
+ Instead of having to explicitly include your module:
292
+
293
+ ```ruby
294
+ class Poro
295
+ include Q::Methods::FooQueue
296
+ end
297
+ ```
298
+
299
+ ## Enqueue Background Tasks
300
+
301
+ Once your task is defined correctly in the code you need a way to call it. Our ficticious `FooQueue` library allows you to process our `SendEmail` task that we defined previously in the background like this:
302
+
303
+ ```ruby
304
+ user = User.first
305
+ FooQueue.enqueue(Poro::SendEmail, user.id, 'open')
306
+ ```
307
+
308
+ We must be able to acomplish this same thing in our `QueueMethod.call()` method call:
309
+
310
+
311
+ ```ruby
312
+ module Q::Methods::FooQueue
313
+ include Q::Methods::Base
314
+
315
+ class QueueMethod
316
+ def self.call(options = {})
317
+ end
318
+ end
319
+ end
320
+ ```
321
+
322
+ Here options are:
323
+
324
+ ```ruby
325
+ options[:base] # this is the class/module that your code is being included into
326
+ options[:queue_name] # in our example this would be :send_email
327
+ options[:queue_klass_name] # in our example this would be SendEmail
328
+ ```
329
+
330
+ ```ruby
331
+ module Q::Methods::FooQueue
332
+ include Q::Methods::Base
333
+
334
+ class QueueMethod
335
+ def self.call(options = {})
336
+ base = options[:base]
337
+ queue_name = options[:queue_name]
338
+ queue_klass_name = options[:queue_klass_name]
339
+ end
340
+ end
341
+ end
342
+ ```
343
+
344
+ To enqueue our job we'll need the klass reference not just the name:
345
+
346
+ ```ruby
347
+ class QueueMethod
348
+ def self.call(options = {})
349
+ base = options[:base]
350
+ queue_name = options[:queue_name]
351
+ queue_klass_name = options[:queue_klass_name]
352
+
353
+ queue_klass = base.const_get(queue_klass_name)
354
+ end
355
+ end
356
+ ```
357
+
358
+ Now you need to define a method on our queue object using `define_singleton_method`:
359
+
360
+ ```ruby
361
+ class QueueMethod
362
+ def self.call(options = {})
363
+ base = options[:base]
364
+ queue_name = options[:queue_name]
365
+ queue_klass_name = options[:queue_klass_name]
366
+
367
+ queue_klass = base.const_get(queue_klass_name)
368
+
369
+ base.queue.define_singleton_method(queue_name) do |*args|
370
+
371
+ end
372
+ end
373
+ end
374
+ ```
375
+
376
+ The code we want to emulate needs to go in this programatically created method:
377
+
378
+ ```ruby
379
+ class QueueMethod
380
+ def self.call(options = {})
381
+ base = options[:base]
382
+ queue_name = options[:queue_name]
383
+ queue_klass_name = options[:queue_klass_name]
384
+
385
+ queue_klass = base.const_get(queue_klass_name)
386
+
387
+ base.queue.define_singleton_method(queue_name) do |*args|
388
+ FooQueue.enqueue(queue_klass, *args)
389
+ end
390
+ end
391
+ end
392
+ ```
393
+
394
+ Now when a user calls this:
395
+
396
+ ```ruby
397
+ user = User.first
398
+ Poro.queue.send_email(user.id, 'open')
399
+ ```
400
+
401
+ It will be exactly the same as this code:
402
+
403
+ ```
404
+ user = User.first
405
+ FooQueue.enqueue(Poro::SendEmail, user.id, 'open')
406
+ ```
407
+
408
+ Now your code will create a queue, and create background jobs for that queue. All we have left is running your background workers and configuration.
409
+
410
+ ## Convention: JSON
411
+
412
+ While your queue may accept any type of data, not all other queues will. To maximize compatability you should build your `QueueMethod` and `QueueBuild` to only rely on JSON compatible ruby data types (arrays, strings, numbers, hashes). If you don't then you're defeating the whole purpose of writing an integration with `Q` as other users will not be able to use their previously written `Q` compatible apps with your adapter.
413
+
414
+
415
+ ## Running Workers
416
+
417
+ To run the mythical `FooQueue` background workers users must run this rake task:
418
+
419
+ ```ruby
420
+ $ bundle exec rake foo_queue:work VVERBOSE=1 QUEUE=*
421
+ ```
422
+
423
+ The task is `foo_queue:work` and `VVERBOSE` is an environment variable specifying the log level, and `QUEUE` is an environment variable specifying which queues to run, in our case we want to run all of them.
424
+
425
+ We must call this programmatically in our module with the help of our `BuildTask` constant:
426
+
427
+ ```ruby
428
+ module Q::Methods::FooQueue
429
+ include Q::Methods::Base
430
+
431
+ class QueueTask
432
+ def self.call(*rake_args)
433
+ end
434
+ end
435
+ end
436
+ ```
437
+
438
+ Here `*rake_args` are any arguments that may need to be passed into the task. In our example above we're not using any, but your non-fictitious queuing library might:
439
+
440
+ ```ruby
441
+ $ rake your_queue:work[1,2,3]
442
+ ```
443
+
444
+