backburner 0.3.4 → 0.4.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -1,6 +1,12 @@
1
1
  # CHANGELOG
2
2
 
3
- ## Version 0.3.5 (Unreleased)
3
+ ## Version 0.4.0 (June 28 2013)
4
+
5
+ NOTE: This is the start of working with @bradgessler to improve backburner and merge with quebert
6
+
7
+ * NEW #26 #27 Remove need for Queue mixin, allow plain ruby objects
8
+ * NEW Default all jobs to a single general queue rather than separate queues
9
+ * NEW Add support for named priorities, allowing shorthand names for priority values
4
10
 
5
11
  ## Version 0.3.4 (April 23 2013)
6
12
 
data/README.md CHANGED
@@ -10,7 +10,7 @@ If you want to use beanstalk for your job processing, consider using Backburner.
10
10
  Backburner is heavily inspired by Resque and DelayedJob. Backburner stores all jobs as simple JSON message payloads.
11
11
  Persistent queues are supported when beanstalkd persistence mode is enabled.
12
12
 
13
- Backburner supports multiple queues, job priorities, delays, and timeouts. In addition,
13
+ Backburner supports multiple queues, job priorities, delays, and timeouts. In addition,
14
14
  Backburner has robust support for retrying failed jobs, handling error cases,
15
15
  custom logging, and extensible plugin hooks.
16
16
 
@@ -89,7 +89,7 @@ Backburner is extremely simple to setup. Just configure basic settings for backb
89
89
  ```ruby
90
90
  Backburner.configure do |config|
91
91
  config.beanstalk_url = ["beanstalk://127.0.0.1", "..."]
92
- config.tube_namespace = "some-app-production"
92
+ config.tube_namespace = "some.app.production"
93
93
  config.on_error = lambda { |e| puts e }
94
94
  config.max_job_retries = 3 # default 0 retries
95
95
  config.retry_delay = 2 # default 5 seconds
@@ -97,37 +97,67 @@ Backburner.configure do |config|
97
97
  config.respond_timeout = 120
98
98
  config.default_worker = Backburner::Workers::Simple
99
99
  config.logger = Logger.new(STDOUT)
100
+ config.primary_queue = "backburner-jobs"
101
+ config.priority_labels = { :custom => 50, :useless => 1000 }
100
102
  end
101
103
  ```
102
104
 
103
105
  The key options available are:
104
106
 
105
- | Option | Description |
106
- | ------- | ------------------------------- |
107
- | `beanstalk_url` | Address such as 'beanstalk://127.0.0.1' or an array of addresses. |
108
- | `tube_namespace` | Prefix used for all tubes related to this backburner queue. |
109
- | `on_error` | Lambda invoked with the error whenever any job in the system fails. |
110
- | `default_worker` | Worker class that will be used if no other worker is specified. |
111
- | `max_job_retries`| Integer defines how many times to retry a job before burying. |
112
- | `retry_delay` | Integer defines the base time to wait (in secs) between job retries. |
113
- | `logger` | Logger recorded to when backburner wants to report info or errors. |
107
+ | Option | Description |
108
+ | ----------------- | ------------------------------- |
109
+ | `beanstalk_url` | Address such as 'beanstalk://127.0.0.1' or an array of addresses. |
110
+ | `tube_namespace` | Prefix used for all tubes related to this backburner queue. |
111
+ | `on_error` | Lambda invoked with the error whenever any job in the system fails. |
112
+ | `default_worker` | Worker class that will be used if no other worker is specified. |
113
+ | `max_job_retries` | Integer defines how many times to retry a job before burying. |
114
+ | `retry_delay` | Integer defines the base time to wait (in secs) between job retries. |
115
+ | `logger` | Logger recorded to when backburner wants to report info or errors. |
116
+ | `primary_queue` | Primary queue used for a job when an alternate queue is not given. |
117
+ | `priority_labels` | Hash of named priority definitions for your app. |
118
+
119
+ ## Breaking Changes
120
+
121
+ Since **v0.4.0**: Jobs used to be placed into default queues based on the name of the class enqueuing i.e NewsletterJob would
122
+ be put into a 'newsletter-job' queue. After 0.4.0, all jobs are placed into a primary queue named "my.app.namespace.backburner-jobs"
123
+ unless otherwise specified.
114
124
 
115
125
  ## Usage
116
126
 
117
- Backburner allows you to create jobs and place them on a beanstalk queue, and later pull those jobs off the queue and
118
- process them asynchronously.
127
+ Backburner allows you to create jobs and place them onto any number of beanstalk tubes, and later pull those jobs off the tubes and
128
+ process them asynchronously with a worker.
119
129
 
120
130
  ### Enqueuing Jobs ###
121
131
 
122
- At the core, Backburner is about jobs that can be processed. Jobs are simple ruby objects with a method defined named `perform`.
132
+ At the core, Backburner is about jobs that can be processed asynchronously. Jobs are simple ruby objects which respond to `perform`.
133
+
134
+ Job objects are queued as JSON onto a tube to be later processed by a worker. Here's an example:
135
+
136
+ ```ruby
137
+ class NewsletterJob
138
+ # required
139
+ def self.perform(email, body)
140
+ NewsletterMailer.deliver_text_to_email(email, body)
141
+ end
142
+
143
+ # optional, defaults to 'backburner-jobs' tube
144
+ def self.queue
145
+ "newsletter-sender"
146
+ end
147
+
148
+ # optional, defaults to default_priority
149
+ def self.queue_priority
150
+ 1000 # most urgent priority is 0
151
+ end
152
+ end
153
+ ```
123
154
 
124
- Any object which responds to `perform` can be queued as a job. Job objects are queued as JSON to be later processed by a task runner.
125
- Here's an example:
155
+ You can include the optional `Backburner::Queue` module so you can easily specify queue settings for this job:
126
156
 
127
157
  ```ruby
128
158
  class NewsletterJob
129
159
  include Backburner::Queue
130
- queue "newsletter" # defaults to 'newsletter-job'
160
+ queue "newsletter-sender" # defaults to 'backburner-jobs' tube
131
161
  queue_priority 1000 # most urgent priority is 0
132
162
 
133
163
  def self.perform(email, body)
@@ -136,7 +166,6 @@ class NewsletterJob
136
166
  end
137
167
  ```
138
168
 
139
- Notice that you can include the optional `Backburner::Queue` module so you can specify a `queue` name for this job.
140
169
  Jobs can be enqueued with:
141
170
 
142
171
  ```ruby
@@ -144,8 +173,8 @@ Backburner.enqueue NewsletterJob, 'foo@admin.com', 'lorem ipsum...'
144
173
  ```
145
174
 
146
175
  `Backburner.enqueue` accepts first a ruby object that supports `perform` and then a series of parameters
147
- to that object's `perform` method. The queue name used by default is the normalized class name (i.e `{namespace}.newsletter-job`)
148
- if not otherwise specified.
176
+ to that object's `perform` method. The queue name used by default is `{namespace}.backburner-jobs`
177
+ unless otherwise specified.
149
178
 
150
179
  ### Simple Async Jobs ###
151
180
 
@@ -168,21 +197,21 @@ class User
168
197
  end
169
198
  end
170
199
 
171
- # Async works for instance methods on a persisted model
200
+ # Async works for instance methods on a persisted object with an `id`
172
201
  @user = User.first
173
202
  @user.async(:ttr => 100, :queue => "activate").activate(@device.id)
174
- # ..as well as for class methods
203
+ # ..and for class methods
175
204
  User.async(:pri => 100, :delay => 10.seconds).reset_password(@user.id)
176
205
  ```
177
206
 
178
- This will automatically enqueue a job for that user record that will run `activate` with the specified argument.
207
+ This automatically enqueues a job for that user record that will run `activate` with the specified argument.
179
208
  Note that you can set the queue name and queue priority at the class level and
180
209
  you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`.
181
- The queue name used by default is the normalized class name (i.e `{namespace}.user`) if not otherwise specified.
210
+ The queue name used by default is `{namespace}.backburner-jobs` if not otherwise specified.
182
211
 
183
212
  ### Working Jobs
184
213
 
185
- Backburner workers are processes that run forever handling jobs that get reserved. Starting a worker in ruby code is simple:
214
+ Backburner workers are processes that run forever handling jobs that are reserved from the queue. Starting a worker in ruby code is simple:
186
215
 
187
216
  ```ruby
188
217
  Backburner.work
@@ -191,7 +220,7 @@ Backburner.work
191
220
  This will process jobs in all queues but you can also restrict processing to specific queues:
192
221
 
193
222
  ```ruby
194
- Backburner.work('newsletter_sender')
223
+ Backburner.work('newsletter-sender,push-notifier')
195
224
  ```
196
225
 
197
226
  The Backburner worker also exists as a rake task:
@@ -203,13 +232,13 @@ require 'backburner/tasks'
203
232
  so you can run:
204
233
 
205
234
  ```
206
- $ QUEUES=newsletter-sender,push-message rake backburner:work
235
+ $ QUEUES=newsletter-sender,push-notifier rake backburner:work
207
236
  ```
208
237
 
209
238
  You can also run the backburner binary for a convenient worker:
210
239
 
211
240
  ```
212
- bundle exec backburner newsletter-sender,push-message -d -P /var/run/backburner.pid -l /var/log/backburner.log
241
+ bundle exec backburner newsletter-sender,push-notifier -d -P /var/run/backburner.pid -l /var/log/backburner.log
213
242
  ```
214
243
 
215
244
  This will daemonize the worker and store the pid and logs automatically.
@@ -223,7 +252,7 @@ example from above. We'll run the following code to create a job:
223
252
  User.async.reset_password(@user.id)
224
253
  ```
225
254
 
226
- The following JSON will be stored in the `{namespace}.user` queue:
255
+ The following JSON will be put on the `{namespace}.backburner-jobs` queue:
227
256
 
228
257
  ``` javascript
229
258
  {
@@ -252,6 +281,31 @@ would be stored as:
252
281
  Since all jobs are persisted in JSON, your jobs must only accept arguments that can be encoded into that format.
253
282
  This is why our examples use object IDs instead of passing around objects.
254
283
 
284
+ ### Named Priorities
285
+
286
+ As of v0.4.0, Backburner has support for named priorities. beanstalkd priorities are numerical but
287
+ backburner supports a mapping between a word and a numerical value. The following priorities are
288
+ available by default: `high` is 0, `medium` is 100, and `low` is 200.
289
+
290
+ Priorities can be customized with:
291
+
292
+ ```ruby
293
+ Backburner.configure do |config|
294
+ config.priority_labels = { :custom => 50, :useful => 5 }
295
+ # or append to default priorities with
296
+ # config.priority_labels = Backburner::Configuration::PRIORITY_LABELS.merge(:foo => 5)
297
+ end
298
+ ```
299
+
300
+ and then these aliases can be used anywhere that a numerical value can:
301
+
302
+ ```ruby
303
+ Backburner::Worker.enqueue NewsletterJob, ["foo", "bar"], :pri => :custom
304
+ User.async(:pri => :useful, :delay => 10.seconds).reset_password(@user.id)
305
+ ```
306
+
307
+ Using named priorities can greatly simplify priority management.
308
+
255
309
  ### Processing Strategies
256
310
 
257
311
  In Backburner, there are several different strategies for processing jobs
@@ -276,7 +330,7 @@ end
276
330
  or determine the worker on the fly when invoking `work`:
277
331
 
278
332
  ```ruby
279
- Backburner.work('newsletter_sender', :worker => Backburner::Workers::ThreadsOnFork)
333
+ Backburner.work('newsletter-sender', :worker => Backburner::Workers::ThreadsOnFork)
280
334
  ```
281
335
 
282
336
  or through associated rake tasks with:
@@ -295,7 +349,7 @@ If you are interested in helping out, please let us know.
295
349
  Workers can be easily restricted to processing only a specific set of queues as shown above. However, if you want a worker to
296
350
  process **all** queues instead, then you can leave the queue list blank.
297
351
 
298
- When you execute a worker without queues specified, any queue for a known job queue class with `include Backburner::Queue` will be processed.
352
+ When you execute a worker without any queues specified, queues for known job queue class with `include Backburner::Queue` will be processed.
299
353
  To access the list of known queue classes, you can use:
300
354
 
301
355
  ```ruby
@@ -310,7 +364,7 @@ queues processed when none are specified. To do this, you can use the `default_q
310
364
  Backburner.default_queues.concat(["foo", "bar"])
311
365
  ```
312
366
 
313
- This will ensure that the _foo_ and _bar_ queues are processed by default. You can also add job queue names:
367
+ This will ensure that the _foo_ and _bar_ queues are processed by any default workers. You can also add job queue names with:
314
368
 
315
369
  ```ruby
316
370
  Backburner.default_queues << NewsletterJob.queue
@@ -320,7 +374,7 @@ The `default_queues` stores the specific list of queues that should be processed
320
374
 
321
375
  ### Failures
322
376
 
323
- When a job fails in backburner (usually because an exception was raised), the job will be released
377
+ When a job fails in backburner (usually because an exception was raised), the job will be released
324
378
  and retried again (with progressive delays in between) until the `max_job_retries` configuration is reached.
325
379
 
326
380
  ```ruby
@@ -362,8 +416,8 @@ Be sure to check logs whenever things do not seem to be processing.
362
416
  ### Hooks
363
417
 
364
418
  Backburner is highly extensible and can be tailored to your needs by using various hooks that
365
- can be triggered across the job processing lifecycle.
366
- Often using hooks is much easier then trying to monkey patch the externals.
419
+ can be triggered across the job processing lifecycle.
420
+ Often using hooks is much easier then trying to monkey patch the externals.
367
421
 
368
422
  Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
369
423
 
@@ -381,7 +435,7 @@ and then you can start the rake task with:
381
435
 
382
436
  ```bash
383
437
  $ rake backburner:work
384
- $ QUEUES=newsletter-sender,push-message rake backburner:work
438
+ $ QUEUES=newsletter-sender,push-notifier rake backburner:work
385
439
  ```
386
440
 
387
441
  The best way to deploy these rake tasks is using a monitoring library. We suggest [God](https://github.com/mojombo/god/)
@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
15
15
  s.require_paths = ["lib"]
16
16
  s.version = Backburner::VERSION
17
17
 
18
- s.add_runtime_dependency 'beaneater', '~> 0.3.0'
18
+ s.add_runtime_dependency 'beaneater', '~> 0.3.1'
19
19
  s.add_runtime_dependency 'dante', '~> 0.1.5'
20
20
 
21
21
  s.add_development_dependency 'rake'
@@ -1,5 +1,7 @@
1
1
  module Backburner
2
2
  class Configuration
3
+ PRIORITY_LABELS = { :high => 0, :medium => 100, :low => 200 }
4
+
3
5
  attr_accessor :beanstalk_url # beanstalk url connection
4
6
  attr_accessor :tube_namespace # namespace prefix for every queue
5
7
  attr_accessor :default_priority # default job priority
@@ -10,6 +12,8 @@ module Backburner
10
12
  attr_accessor :default_queues # default queues
11
13
  attr_accessor :logger # logger
12
14
  attr_accessor :default_worker # default worker class
15
+ attr_accessor :primary_queue # the general queue
16
+ attr_accessor :priority_labels # priority labels
13
17
 
14
18
  def initialize
15
19
  @beanstalk_url = "beanstalk://localhost"
@@ -22,6 +26,8 @@ module Backburner
22
26
  @default_queues = []
23
27
  @logger = nil
24
28
  @default_worker = Backburner::Workers::Simple
29
+ @primary_queue = "backburner-jobs"
30
+ @priority_labels = PRIORITY_LABELS
25
31
  end
26
32
  end # Configuration
27
33
  end # Backburner
@@ -89,13 +89,32 @@ module Backburner
89
89
  tube
90
90
  elsif tube.respond_to?(:queue) # use queue name
91
91
  tube.queue
92
- elsif tube.is_a?(Class) # no queue name, use job_class
93
- tube.name
92
+ elsif tube.is_a?(Class) # no queue name, use default
93
+ queue_config.primary_queue # tube.name
94
94
  else # turn into a string
95
95
  tube.to_s
96
96
  end
97
97
  [prefix.gsub(/\.$/, ''), dasherize(queue_name).gsub(/^#{prefix}/, '')].join(".").gsub(/\.+/, '.')
98
98
  end
99
99
 
100
- end
101
- end
100
+ # Resolves job priority based on the value given. Can be integer, a class or nothing
101
+ #
102
+ # @example
103
+ # resolve_priority(1000) => 1000
104
+ # resolve_priority(FooBar) => <queue priority>
105
+ # resolve_priority(nil) => <default priority>
106
+ #
107
+ def resolve_priority(pri)
108
+ if pri.respond_to?(:queue_priority)
109
+ resolve_priority(pri.queue_priority)
110
+ elsif pri.is_a?(String) || pri.is_a?(Symbol) # named priority
111
+ resolve_priority(Backburner.configuration.priority_labels[pri.to_sym])
112
+ elsif pri.is_a?(Fixnum) # numerical
113
+ pri
114
+ else # default
115
+ Backburner.configuration.default_priority
116
+ end
117
+ end
118
+
119
+ end # Helpers
120
+ end # Backburner
@@ -1,51 +1,53 @@
1
1
  module Backburner
2
- module Hooks
3
- # Triggers all method hooks that match the given event type with specified arguments.
4
- #
5
- # @example
6
- # invoke_hook_events(:before_enqueue, 'some', 'args')
7
- # invoke_hook_events(:after_perform, 5)
8
- #
9
- def invoke_hook_events(event, *args)
10
- res = find_hook_events(event).map { |e| send(e, *args) }
11
- return false if res.any? { |result| result == false }
12
- res
13
- end
2
+ class Hooks
3
+ class << self
4
+ # Triggers all method hooks that match the given event type with specified arguments.
5
+ #
6
+ # @example
7
+ # invoke_hook_events(:before_enqueue, 'some', 'args')
8
+ # invoke_hook_events(:after_perform, 5)
9
+ #
10
+ def invoke_hook_events(job, event, *args)
11
+ res = find_hook_events(job, event).map { |e| job.send(e, *args) }
12
+ return false if res.any? { |result| result == false }
13
+ res
14
+ end
14
15
 
15
- # Triggers all method hooks that match given around event type. Used for 'around' hooks
16
- # that stack over the original task cumulatively onto one another.
17
- #
18
- # The final block will be the one that actually invokes the
19
- # original task after calling all other around blocks.
20
- #
21
- # @example
22
- # around_hook_events(:around_perform) { job.perform }
23
- #
24
- def around_hook_events(event, *args, &block)
25
- raise "Please pass a block to hook events!" unless block_given?
26
- around_hooks = find_hook_events(event).reverse
27
- aggregate_filter = Proc.new { |&blk| blk.call }
28
- around_hooks.each do |ah|
29
- prior_around_filter = aggregate_filter
30
- aggregate_filter = Proc.new do |&blk|
31
- method(ah).call(*args) do
32
- prior_around_filter.call(&blk)
16
+ # Triggers all method hooks that match given around event type. Used for 'around' hooks
17
+ # that stack over the original task cumulatively onto one another.
18
+ #
19
+ # The final block will be the one that actually invokes the
20
+ # original task after calling all other around blocks.
21
+ #
22
+ # @example
23
+ # around_hook_events(:around_perform) { job.perform }
24
+ #
25
+ def around_hook_events(job, event, *args, &block)
26
+ raise "Please pass a block to hook events!" unless block_given?
27
+ around_hooks = find_hook_events(job, event).reverse
28
+ aggregate_filter = Proc.new { |&blk| blk.call }
29
+ around_hooks.each do |ah|
30
+ prior_around_filter = aggregate_filter
31
+ aggregate_filter = Proc.new do |&blk|
32
+ job.method(ah).call(*args) do
33
+ prior_around_filter.call(&blk)
34
+ end
33
35
  end
34
36
  end
37
+ aggregate_filter.call(&block)
35
38
  end
36
- aggregate_filter.call(&block)
37
- end
38
39
 
39
- protected
40
+ protected
40
41
 
41
- # Returns all methods that match given hook type
42
- #
43
- # @example
44
- # find_hook_events(:before_enqueue)
45
- # # => ['before_enqueue_foo', 'before_enqueue_bar']
46
- #
47
- def find_hook_events(event)
48
- (self.methods - Object.methods).grep(/^#{event}/).sort
42
+ # Returns all methods that match given hook type
43
+ #
44
+ # @example
45
+ # find_hook_events(:before_enqueue)
46
+ # # => ['before_enqueue_foo', 'before_enqueue_bar']
47
+ #
48
+ def find_hook_events(job, event)
49
+ (job.methods - Object.methods).grep(/^#{event}/).sort
50
+ end
49
51
  end
50
52
  end # Hooks
51
53
  end # Backburner
@@ -22,6 +22,7 @@ module Backburner
22
22
  @task = task
23
23
  @body = task.body.is_a?(Hash) ? task.body : JSON.parse(task.body)
24
24
  @name, @args = body["class"], body["args"]
25
+ @hooks = Backburner::Hooks
25
26
  rescue => ex # Job was not valid format
26
27
  self.bury
27
28
  raise JobFormatInvalid, "Job body could not be parsed: #{ex.inspect}"
@@ -41,17 +42,17 @@ module Backburner
41
42
  #
42
43
  def process
43
44
  # Invoke before hook and stop if false
44
- res = job_class.invoke_hook_events(:before_perform, *args)
45
+ res = @hooks.invoke_hook_events(job_class, :before_perform, *args)
45
46
  return false unless res
46
47
  # Execute the job
47
- job_class.around_hook_events(:around_perform, *args) do
48
+ @hooks.around_hook_events(job_class, :around_perform, *args) do
48
49
  timeout_job_after(task.ttr - 1) { job_class.perform(*args) }
49
50
  end
50
51
  task.delete
51
52
  # Invoke after perform hook
52
- job_class.invoke_hook_events(:after_perform, *args)
53
+ @hooks.invoke_hook_events(job_class, :after_perform, *args)
53
54
  rescue => e
54
- job_class.invoke_hook_events(:on_failure, e, *args)
55
+ @hooks.invoke_hook_events(job_class, :on_failure, e, *args)
55
56
  raise e
56
57
  end
57
58
 
@@ -1,8 +1,6 @@
1
1
  module Backburner
2
2
  module Queue
3
3
  def self.included(base)
4
- base.send(:extend, Backburner::Helpers)
5
- base.send(:extend, Backburner::Hooks)
6
4
  base.extend ClassMethods
7
5
  Backburner::Worker.known_queue_classes << base
8
6
  end
@@ -18,7 +16,7 @@ module Backburner
18
16
  if name
19
17
  @queue_name = name
20
18
  else # accessor
21
- @queue_name || dasherize(self.name)
19
+ @queue_name || Backburner.configuration.primary_queue
22
20
  end
23
21
  end
24
22
 
@@ -1,3 +1,3 @@
1
1
  module Backburner
2
- VERSION = "0.3.4"
2
+ VERSION = "0.4.0"
3
3
  end
@@ -24,15 +24,15 @@ module Backburner
24
24
  # Backburner::Worker.enqueue NewsletterSender, [self.id, user.id], :ttr => 1000
25
25
  #
26
26
  def self.enqueue(job_class, args=[], opts={})
27
- pri = opts[:pri] || job_class.queue_priority || Backburner.configuration.default_priority
27
+ pri = resolve_priority(opts[:pri] || job_class)
28
28
  delay = [0, opts[:delay].to_i].max
29
29
  ttr = opts[:ttr] || Backburner.configuration.respond_timeout
30
30
  tube = connection.tubes[expand_tube_name(opts[:queue] || job_class)]
31
- res = job_class.invoke_hook_events(:before_enqueue, *args)
31
+ res = Backburner::Hooks.invoke_hook_events(job_class, :before_enqueue, *args)
32
32
  return false unless res # stop if hook is false
33
33
  data = { :class => job_class.name, :args => args }
34
34
  tube.put data.to_json, :pri => pri, :delay => delay, :ttr => ttr
35
- job_class.invoke_hook_events(:after_enqueue, *args)
35
+ Backburner::Hooks.invoke_hook_events(job_class, :after_enqueue, *args)
36
36
  return true
37
37
  end
38
38
 
@@ -143,7 +143,7 @@ module Backburner
143
143
  def all_existing_queues
144
144
  known_queues = Backburner::Worker.known_queue_classes.map(&:queue)
145
145
  existing_tubes = self.connection.tubes.all.map(&:name).select { |tube| tube =~ /^#{queue_config.tube_namespace}/ }
146
- known_queues + existing_tubes
146
+ known_queues + existing_tubes + [queue_config.primary_queue]
147
147
  end
148
148
 
149
149
  # Returns a reference to the beanstalk connection
@@ -171,7 +171,7 @@ module Backburner
171
171
  tube_names = Array(tube_names).compact if tube_names && Array(tube_names).compact.size > 0
172
172
  tube_names = nil if tube_names && tube_names.compact.empty?
173
173
  tube_names ||= Backburner.default_queues.any? ? Backburner.default_queues : all_existing_queues
174
- Array(tube_names)
174
+ Array(tube_names).uniq
175
175
  end
176
176
 
177
177
  # Registers signal handlers TERM and INT to trigger
@@ -1,21 +1,18 @@
1
1
  require File.expand_path('../test_helper', __FILE__)
2
2
 
3
- class AsyncUser; def self.invoke_hook_events(*args); true; end; end
3
+ class AsyncUser; end
4
4
 
5
5
  describe "Backburner::AsyncProxy class" do
6
6
  before do
7
7
  Backburner.default_queues.clear
8
- end
9
-
10
- after do
11
- clear_jobs!("async-user")
8
+ clear_jobs!(Backburner.configuration.primary_queue)
12
9
  end
13
10
 
14
11
  describe "for method_missing enqueue" do
15
12
  should "enqueue job onto worker with no args" do
16
13
  @async = Backburner::AsyncProxy.new(AsyncUser, 10, :pri => 1000, :ttr => 100)
17
14
  @async.foo
18
- job, body = pop_one_job("async-user")
15
+ job, body = pop_one_job
19
16
  assert_equal "AsyncUser", body["class"]
20
17
  assert_equal [10, "foo"], body["args"]
21
18
  assert_equal 100, job.ttr
@@ -26,7 +23,7 @@ describe "Backburner::AsyncProxy class" do
26
23
  should "enqueue job onto worker with args" do
27
24
  @async = Backburner::AsyncProxy.new(AsyncUser, 10, :pri => 1000, :ttr => 100)
28
25
  @async.bar(1, 2, 3)
29
- job, body = pop_one_job("async-user")
26
+ job, body = pop_one_job
30
27
  assert_equal "AsyncUser", body["class"]
31
28
  assert_equal [10, "bar", 1, 2, 3], body["args"]
32
29
  assert_equal 100, job.ttr
@@ -15,8 +15,6 @@ end
15
15
 
16
16
 
17
17
  class HookedObjectAfterEnqueueFail
18
- extend Backburner::Hooks
19
-
20
18
  def self.after_enqueue_abe(*args)
21
19
  puts "!!after_enqueue_foo!! #{args.inspect}"
22
20
  end
@@ -44,8 +42,6 @@ class HookedObjectBeforePerformFail
44
42
  end
45
43
 
46
44
  class HookedObjectAfterPerformFail
47
- extend Backburner::Hooks
48
-
49
45
  def self.after_perform_abe(*args)
50
46
  puts "!!after_perform_foo!! #{args.inspect}"
51
47
  end
@@ -56,8 +52,6 @@ class HookedObjectAfterPerformFail
56
52
  end
57
53
 
58
54
  class HookedObjectJobFailure
59
- extend Backburner::Hooks
60
-
61
55
  def self.foo(x)
62
56
  raise HookFailError, "HookedObjectJobFailure on foo!"
63
57
  end
@@ -11,6 +11,7 @@ end
11
11
 
12
12
  class TestJobFork
13
13
  include Backburner::Queue
14
+ queue "test-job-fork"
14
15
  queue_priority 1000
15
16
  def self.perform(x, y)
16
17
  Backburner::Workers::ThreadsOnFork.enqueue ResponseJob, [{
@@ -21,6 +22,7 @@ end
21
22
 
22
23
  class TestFailJobFork
23
24
  include Backburner::Queue
25
+ queue "test-fail-job-fork"
24
26
  def self.perform(x, y)
25
27
  Backburner::Workers::ThreadsOnFork.enqueue ResponseJob, [{
26
28
  :worker_raise => true
@@ -1,9 +1,14 @@
1
1
  $worker_test_count = 0
2
2
  $worker_success = false
3
3
 
4
+ class TestPlainJob
5
+ def self.queue; "test-plain"; end
6
+ def self.perform(x, y); $worker_test_count += x + y + 1; end
7
+ end
8
+
4
9
  class TestJob
5
10
  include Backburner::Queue
6
- queue_priority 1000
11
+ queue_priority :medium
7
12
  def self.perform(x, y); $worker_test_count += x + y; end
8
13
  end
9
14
 
@@ -53,7 +53,7 @@ describe "Backburner::Helpers module" do
53
53
  end # config
54
54
 
55
55
  describe "for expand_tube_name method" do
56
- before { Backburner.expects(:configuration).returns(stub(:tube_namespace => "test.foo.job.")) }
56
+ before { Backburner.stubs(:configuration).returns(stub(:tube_namespace => "test.foo.job.", :primary_queue => "backburner-jobs")) }
57
57
 
58
58
  it "supports base strings" do
59
59
  assert_equal "test.foo.job.email/send-news", expand_tube_name("email/send_news")
@@ -73,7 +73,55 @@ describe "Backburner::Helpers module" do
73
73
  end # queue names
74
74
 
75
75
  it "supports class names" do
76
- assert_equal "test.foo.job.runtime-error", expand_tube_name(RuntimeError)
76
+ assert_equal "test.foo.job.backburner-jobs", expand_tube_name(RuntimeError)
77
77
  end # class names
78
78
  end # expand_tube_name
79
+
80
+ describe "for resolve_priority method" do
81
+ before { Backburner.configure { |config| config.default_priority = 1000 } }
82
+ after { Backburner.configure { |config| config.priority_labels = Backburner::Configuration::PRIORITY_LABELS } }
83
+
84
+ it "supports fix num priority" do
85
+ assert_equal 500, resolve_priority(500)
86
+ end
87
+
88
+ it "supports baked in priority alias" do
89
+ assert_equal 200, resolve_priority(:low)
90
+ assert_equal 0, resolve_priority(:high)
91
+ end
92
+
93
+ it "supports custom priority alias" do
94
+ Backburner.configure { |config| config.priority_labels = { :foo => 5 } }
95
+ assert_equal 5, resolve_priority(:foo)
96
+ end
97
+
98
+ it "supports aliased priority alias" do
99
+ Backburner.configure { |config| config.priority_labels = { :foo => 5, :bar => 'foo' } }
100
+ assert_equal 5, resolve_priority(:bar)
101
+ end
102
+
103
+ it "supports classes which respond to queue_priority" do
104
+ job = stub(:queue_priority => 600)
105
+ assert_equal 600, resolve_priority(job)
106
+ end
107
+
108
+ it "supports classes which respond to queue_priority with named alias" do
109
+ job = stub(:queue_priority => :low)
110
+ assert_equal 200, resolve_priority(job)
111
+ end
112
+
113
+ it "supports classes which returns null queue_priority" do
114
+ job = stub(:queue_priority => nil)
115
+ assert_equal 1000, resolve_priority(job)
116
+ end
117
+
118
+ it "supports classes which don't respond to queue_priority" do
119
+ job = stub(:fake => true)
120
+ assert_equal 1000, resolve_priority(job)
121
+ end
122
+
123
+ it "supports default pri for null values" do
124
+ assert_equal 1000, resolve_priority(nil)
125
+ end
126
+ end # resolve_priority
79
127
  end
@@ -4,19 +4,20 @@ require File.expand_path('../fixtures/hooked', __FILE__)
4
4
  describe "Backburner::Hooks module" do
5
5
  before do
6
6
  $hooked_fail_count = 0
7
+ @hooks = Backburner::Hooks
7
8
  end
8
9
 
9
10
  describe "for invoke_hook_events method" do
10
11
  describe "with before_enqueue" do
11
12
  it "should support successful invocation" do
12
- out = silenced { @res = HookedObjectSuccess.invoke_hook_events(:before_enqueue, 5, 6) }
13
+ out = silenced { @res = @hooks.invoke_hook_events(HookedObjectSuccess, :before_enqueue, 5, 6) }
13
14
  assert_equal [nil, nil], @res
14
15
  assert_match /!!before_enqueue_foo!! \[5\, 6\]/, out
15
16
  assert_match /!!before_enqueue_bar!! \[5\, 6\]/, out
16
17
  end
17
18
 
18
19
  it "should support fail case" do
19
- out = silenced { @res = HookedObjectBeforeEnqueueFail.invoke_hook_events(:before_enqueue, 5, 6) }
20
+ out = silenced { @res = @hooks.invoke_hook_events(HookedObjectBeforeEnqueueFail, :before_enqueue, 5, 6) }
20
21
  assert_equal false, @res
21
22
  assert_match /!!before_enqueue_foo!! \[5\, 6\]/, out
22
23
  end
@@ -24,26 +25,26 @@ describe "Backburner::Hooks module" do
24
25
 
25
26
  describe "with after_enqueue" do
26
27
  it "should support successful invocation" do
27
- out = silenced { HookedObjectSuccess.invoke_hook_events(:after_enqueue, 7, 8) }
28
+ out = silenced { @hooks.invoke_hook_events(HookedObjectSuccess, :after_enqueue, 7, 8) }
28
29
  assert_match /!!after_enqueue_foo!! \[7\, 8\]/, out
29
30
  assert_match /!!after_enqueue_bar!! \[7\, 8\]/, out
30
31
  end
31
32
 
32
33
  it "should support fail case" do
33
34
  assert_raises(HookFailError) do
34
- silenced { @res = HookedObjectAfterEnqueueFail.invoke_hook_events(:after_enqueue, 5, 6) }
35
+ silenced { @res = @hooks.invoke_hook_events(HookedObjectAfterEnqueueFail, :after_enqueue, 5, 6) }
35
36
  end
36
37
  end
37
38
  end # after_enqueue
38
39
 
39
40
  describe "with before_perform" do
40
41
  it "should support successful invocation" do
41
- out = silenced { HookedObjectSuccess.invoke_hook_events(:before_perform, 1, 2) }
42
+ out = silenced { @hooks.invoke_hook_events(HookedObjectSuccess, :before_perform, 1, 2) }
42
43
  assert_match /!!before_perform_foo!! \[1\, 2\]/, out
43
44
  end
44
45
 
45
46
  it "should support fail case" do
46
- out = silenced { @res = HookedObjectBeforePerformFail.invoke_hook_events(:before_perform, 5, 6) }
47
+ out = silenced { @res = @hooks.invoke_hook_events(HookedObjectBeforePerformFail, :before_perform, 5, 6) }
47
48
  assert_equal false, @res
48
49
  assert_match /!!before_perform_foo!! \[5\, 6\]/, out
49
50
  end
@@ -51,20 +52,20 @@ describe "Backburner::Hooks module" do
51
52
 
52
53
  describe "with after_perform" do
53
54
  it "should support successful invocation" do
54
- out = silenced { HookedObjectSuccess.invoke_hook_events(:after_perform, 3, 4) }
55
+ out = silenced { @hooks.invoke_hook_events(HookedObjectSuccess, :after_perform, 3, 4) }
55
56
  assert_match /!!after_perform_foo!! \[3\, 4\]/, out
56
57
  end
57
58
 
58
59
  it "should support fail case" do
59
60
  assert_raises(HookFailError) do
60
- silenced { @res = HookedObjectAfterPerformFail.invoke_hook_events(:after_perform, 5, 6) }
61
+ silenced { @res = @hooks.invoke_hook_events(HookedObjectAfterPerformFail, :after_perform, 5, 6) }
61
62
  end
62
63
  end
63
64
  end # after_perform
64
65
 
65
66
  describe "with on_failure" do
66
67
  it "should support successful invocation" do
67
- out = silenced { HookedObjectSuccess.invoke_hook_events(:on_failure, RuntimeError, 10) }
68
+ out = silenced { @hooks.invoke_hook_events(HookedObjectSuccess, :on_failure, RuntimeError, 10) }
68
69
  assert_match /!!on_failure_foo!! RuntimeError \[10\]/, out
69
70
  end
70
71
  end # on_failure
@@ -74,7 +75,7 @@ describe "Backburner::Hooks module" do
74
75
  describe "with around_perform" do
75
76
  it "should support successful invocation" do
76
77
  out = silenced do
77
- HookedObjectSuccess.around_hook_events(:around_perform, 7, 8) {
78
+ @hooks.around_hook_events(HookedObjectSuccess, :around_perform, 7, 8) {
78
79
  puts "!!FIRED!!"
79
80
  }
80
81
  end
@@ -2,7 +2,6 @@ require File.expand_path('../test_helper', __FILE__)
2
2
 
3
3
  module NestedDemo
4
4
  class TestJobC
5
- include Backburner::Queue
6
5
  def self.perform(x); puts "Performed #{x} in #{self}"; end
7
6
  end
8
7
 
@@ -15,7 +15,7 @@ describe "Backburner::Queue module" do
15
15
 
16
16
  describe "for queue method accessor" do
17
17
  it "should return the queue name" do
18
- assert_equal "nested-demo/test-job-a", NestedDemo::TestJobA.queue
18
+ assert_equal Backburner.configuration.primary_queue, NestedDemo::TestJobA.queue
19
19
  end
20
20
  end # queue_name
21
21
 
@@ -84,7 +84,7 @@ class MiniTest::Spec
84
84
  end
85
85
 
86
86
  # pop_one_job(tube_name)
87
- def pop_one_job(tube_name)
87
+ def pop_one_job(tube_name=Backburner.configuration.primary_queue)
88
88
  connection = Backburner::Worker.connection
89
89
  tube_name = [Backburner.configuration.tube_namespace, tube_name].join(".")
90
90
  connection.tubes.watch!(tube_name)
@@ -5,17 +5,36 @@ require File.expand_path('../fixtures/hooked', __FILE__)
5
5
  describe "Backburner::Worker module" do
6
6
  before do
7
7
  Backburner.default_queues.clear
8
+ clear_jobs!(Backburner.configuration.primary_queue)
8
9
  end
9
10
 
10
11
  describe "for enqueue class method" do
11
- it "should support enqueuing job" do
12
+ it "should support enqueuing plain job" do
13
+ Backburner::Worker.enqueue TestPlainJob, [7, 9], :ttr => 100, :pri => 2000
14
+ job, body = pop_one_job("test-plain")
15
+ assert_equal "TestPlainJob", body["class"]
16
+ assert_equal [7, 9], body["args"]
17
+ assert_equal 100, job.ttr
18
+ assert_equal 2000, job.pri
19
+ end # plain
20
+
21
+ it "should support enqueuing job with class queue priority" do
12
22
  Backburner::Worker.enqueue TestJob, [3, 4], :ttr => 100
13
- job, body = pop_one_job("test-job")
23
+ job, body = pop_one_job
24
+ assert_equal "TestJob", body["class"]
25
+ assert_equal [3, 4], body["args"]
26
+ assert_equal 100, job.ttr
27
+ assert_equal 100, job.pri
28
+ end # queue
29
+
30
+ it "should support enqueuing job with specified named priority" do
31
+ Backburner::Worker.enqueue TestJob, [3, 4], :ttr => 100, :pri => 'high'
32
+ job, body = pop_one_job
14
33
  assert_equal "TestJob", body["class"]
15
34
  assert_equal [3, 4], body["args"]
16
35
  assert_equal 100, job.ttr
17
- assert_equal 1000, job.pri
18
- end # simple
36
+ assert_equal 0, job.pri
37
+ end # queue
19
38
 
20
39
  it "should support enqueuing job with custom queue" do
21
40
  Backburner::Worker.enqueue TestJob, [6, 7], :queue => "test.bar", :pri => 5000
@@ -46,8 +46,8 @@ describe "Backburner::Workers::Forking module" do
46
46
  it "should properly retrieve all tubes" do
47
47
  worker = @worker_class.new
48
48
  out = capture_stdout { worker.prepare }
49
- assert_contains worker.tube_names, "demo.test.test-job"
50
- assert_contains @worker_class.connection.tubes.watched.map(&:name), "demo.test.test-job"
49
+ assert_contains worker.tube_names, "demo.test.backburner-jobs"
50
+ assert_contains @worker_class.connection.tubes.watched.map(&:name), "demo.test.backburner-jobs"
51
51
  assert_match /demo\.test\.test-job/, out
52
52
  end # all read
53
53
  end # prepare
@@ -46,8 +46,8 @@ describe "Backburner::Workers::Basic module" do
46
46
  it "should properly retrieve all tubes" do
47
47
  worker = @worker_class.new
48
48
  out = capture_stdout { worker.prepare }
49
- assert_contains worker.tube_names, "demo.test.test-job"
50
- assert_contains @worker_class.connection.tubes.watched.map(&:name), "demo.test.test-job"
49
+ assert_contains worker.tube_names, "demo.test.backburner-jobs"
50
+ assert_contains @worker_class.connection.tubes.watched.map(&:name), "demo.test.backburner-jobs"
51
51
  assert_match /demo\.test\.test-job/, out
52
52
  end # all read
53
53
  end # prepare
@@ -58,6 +58,17 @@ describe "Backburner::Workers::Basic module" do
58
58
  $worker_success = false
59
59
  end
60
60
 
61
+ it "should work a plain enqueued job" do
62
+ clear_jobs!("foo.bar")
63
+ @worker_class.enqueue TestPlainJob, [1, 2], :queue => "foo.bar"
64
+ silenced(2) do
65
+ worker = @worker_class.new('foo.bar')
66
+ worker.prepare
67
+ worker.work_one_job
68
+ end
69
+ assert_equal 4, $worker_test_count
70
+ end # plain enqueue
71
+
61
72
  it "should work an enqueued job" do
62
73
  clear_jobs!("foo.bar")
63
74
  @worker_class.enqueue TestJob, [1, 2], :queue => "foo.bar"
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: backburner
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.3.4
4
+ version: 0.4.0
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2013-04-23 00:00:00.000000000 Z
12
+ date: 2013-06-28 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: beaneater
@@ -18,7 +18,7 @@ dependencies:
18
18
  requirements:
19
19
  - - ~>
20
20
  - !ruby/object:Gem::Version
21
- version: 0.3.0
21
+ version: 0.3.1
22
22
  type: :runtime
23
23
  prerelease: false
24
24
  version_requirements: !ruby/object:Gem::Requirement
@@ -26,7 +26,7 @@ dependencies:
26
26
  requirements:
27
27
  - - ~>
28
28
  - !ruby/object:Gem::Version
29
- version: 0.3.0
29
+ version: 0.3.1
30
30
  - !ruby/object:Gem::Dependency
31
31
  name: dante
32
32
  requirement: !ruby/object:Gem::Requirement
@@ -168,7 +168,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
168
168
  version: '0'
169
169
  segments:
170
170
  - 0
171
- hash: 336951971808813718
171
+ hash: 3112315075043673734
172
172
  required_rubygems_version: !ruby/object:Gem::Requirement
173
173
  none: false
174
174
  requirements:
@@ -177,7 +177,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
177
177
  version: '0'
178
178
  segments:
179
179
  - 0
180
- hash: 336951971808813718
180
+ hash: 3112315075043673734
181
181
  requirements: []
182
182
  rubyforge_project:
183
183
  rubygems_version: 1.8.25