backburner 0.2.5 → 0.2.6
Sign up to get free protection for your applications and to get access to all the features.
- data/CHANGELOG.md +5 -1
- data/README.md +71 -29
- data/backburner.gemspec +1 -1
- data/lib/backburner/version.rb +1 -1
- data/test/test_helper.rb +1 -1
- metadata +5 -4
data/CHANGELOG.md
CHANGED
data/README.md
CHANGED
@@ -47,16 +47,18 @@ because it is an in-memory, event-driven system. Powered by libevent under the h
|
|
47
47
|
it requires zero setup (launch and forget, à la memcached), optional log based persistence, an easily parsed ASCII protocol,
|
48
48
|
and a rich set of tools for job management that go well beyond a simple FIFO work queue.
|
49
49
|
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
59
|
-
|
50
|
+
Beanstalkd supports the following features out of the box:
|
51
|
+
|
52
|
+
| Feature | Description |
|
53
|
+
| ------- | ------------------------------- |
|
54
|
+
| **Parallelized** | Supports multiple work queues created on demand. |
|
55
|
+
| **Reliable** | Beanstalk’s reserve, work, delete cycle ensures reliable processing. |
|
56
|
+
| **Scheduling** | Delay enqueuing jobs by a specified interval to schedule processing later |
|
57
|
+
| **Fast** | Processes thousands of jobs per second without breaking a sweat. |
|
58
|
+
| **Priorities** | Specify priority so important jobs can be processed quickly. |
|
59
|
+
| **Persistence** | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
|
60
|
+
| **Federation** | Horizontal scalability provided through federation by the client. |
|
61
|
+
| **Error Handling** | Bury any job which causes an error for later debugging and inspection.|
|
60
62
|
|
61
63
|
Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
|
62
64
|
In the end, **beanstalk is the ideal job queue** while also being ridiculously easy to install and setup.
|
@@ -100,13 +102,15 @@ end
|
|
100
102
|
|
101
103
|
The key options available are:
|
102
104
|
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
|
105
|
+
| Option | Description |
|
106
|
+
| ------- | ------------------------------- |
|
107
|
+
| `beanstalk_url` | Address such as 'beanstalk://127.0.0.1' or an array of addresses. |
|
108
|
+
| `tube_namespace` | Prefix used for all tubes related to this backburner queue. |
|
109
|
+
| `on_error` | Lambda invoked with the error whenever any job in the system fails. |
|
110
|
+
| `default_worker` | Worker class that will be used if no other worker is specified. |
|
111
|
+
| `max_job_retries`| Integer defines how many times to retry a job before burying. |
|
112
|
+
| `retry_delay` | Integer defines the base time to wait (in secs) between job retries. |
|
113
|
+
| `logger` | Logger recorded to when backburner wants to report info or errors. |
|
110
114
|
|
111
115
|
## Usage
|
112
116
|
|
@@ -210,6 +214,44 @@ bundle exec backburner newsletter-sender,push-message -d -P /var/run/backburner.
|
|
210
214
|
|
211
215
|
This will daemonize the worker and store the pid and logs automatically.
|
212
216
|
|
217
|
+
### Persistence
|
218
|
+
|
219
|
+
Jobs are persisted to queues as JSON objects. Let's take our `User`
|
220
|
+
example from above. We'll run the following code to create a job:
|
221
|
+
|
222
|
+
``` ruby
|
223
|
+
User.async.reset_password(@user.id)
|
224
|
+
```
|
225
|
+
|
226
|
+
The following JSON will be stored in the `{namespace}.user` queue:
|
227
|
+
|
228
|
+
``` javascript
|
229
|
+
{
|
230
|
+
'class': 'User',
|
231
|
+
'args': [nil, 'reset_password', 123]
|
232
|
+
}
|
233
|
+
```
|
234
|
+
|
235
|
+
The first argument is the 'id' of the object in the case of an instance method being async'ed. For example:
|
236
|
+
|
237
|
+
```ruby
|
238
|
+
@device = Device.find(987)
|
239
|
+
@user = User.find(246)
|
240
|
+
@user.async.activate(@device.id)
|
241
|
+
```
|
242
|
+
|
243
|
+
would be stored as:
|
244
|
+
|
245
|
+
``` javascript
|
246
|
+
{
|
247
|
+
'class': 'User',
|
248
|
+
'args': [246, 'activate', 987]
|
249
|
+
}
|
250
|
+
```
|
251
|
+
|
252
|
+
Since all jobs are persisted in JSON, your jobs must only accept arguments that can be encoded into that format.
|
253
|
+
This is why our examples use object IDs instead of passing around objects.
|
254
|
+
|
213
255
|
### Processing Strategies
|
214
256
|
|
215
257
|
In Backburner, there are actually multiple different strategies for processing jobs
|
@@ -267,14 +309,6 @@ Backburner.default_queues << NewsletterJob.queue
|
|
267
309
|
|
268
310
|
The `default_queues` stores the specific list of queues that should be processed by default by a worker.
|
269
311
|
|
270
|
-
### Hooks
|
271
|
-
|
272
|
-
Backburner is highly extensible and can be tailored to your needs by using various hooks that
|
273
|
-
can be triggered across the job processing lifecycle.
|
274
|
-
Often using hooks is much easier then trying to monkey patch the externals.
|
275
|
-
|
276
|
-
Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
|
277
|
-
|
278
312
|
### Failures
|
279
313
|
|
280
314
|
When a job fails in backburner (usually because an exception was raised), the job will be released
|
@@ -316,11 +350,13 @@ end
|
|
316
350
|
|
317
351
|
Be sure to check logs whenever things do not seem to be processing.
|
318
352
|
|
319
|
-
###
|
353
|
+
### Hooks
|
320
354
|
|
321
|
-
|
322
|
-
|
323
|
-
|
355
|
+
Backburner is highly extensible and can be tailored to your needs by using various hooks that
|
356
|
+
can be triggered across the job processing lifecycle.
|
357
|
+
Often using hooks is much easier then trying to monkey patch the externals.
|
358
|
+
|
359
|
+
Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
|
324
360
|
|
325
361
|
### Workers in Production
|
326
362
|
|
@@ -343,6 +379,12 @@ The best way to deploy these rake tasks is using a monitoring library. We sugges
|
|
343
379
|
which watches processes and ensures their stability. A simple God recipe for Backburner can be found in
|
344
380
|
[examples/god](https://github.com/nesquena/backburner/blob/master/examples/god.rb).
|
345
381
|
|
382
|
+
### Web Front-end
|
383
|
+
|
384
|
+
Be sure to check out the Sinatra-powered project [beanstalkd_view](https://github.com/denniskuczynski/beanstalkd_view)
|
385
|
+
by [denniskuczynski](http://github.com/denniskuczynski) which provides an excellent overview of the tubes and
|
386
|
+
jobs processed by your beanstalk workers. An excellent addition to your Backburner setup.
|
387
|
+
|
346
388
|
## Acknowledgements
|
347
389
|
|
348
390
|
* [Nathan Esquenazi](https://github.com/nesquena) - Project maintainer
|
data/backburner.gemspec
CHANGED
@@ -15,7 +15,7 @@ Gem::Specification.new do |s|
|
|
15
15
|
s.require_paths = ["lib"]
|
16
16
|
s.version = Backburner::VERSION
|
17
17
|
|
18
|
-
s.add_runtime_dependency 'beaneater', '~> 0.
|
18
|
+
s.add_runtime_dependency 'beaneater', '~> 0.2.0'
|
19
19
|
s.add_runtime_dependency 'dante', '~> 0.1.5'
|
20
20
|
|
21
21
|
s.add_development_dependency 'rake'
|
data/lib/backburner/version.rb
CHANGED
data/test/test_helper.rb
CHANGED
@@ -88,7 +88,7 @@ class MiniTest::Spec
|
|
88
88
|
tube_name = [Backburner.configuration.tube_namespace, tube_name].join(".")
|
89
89
|
connection.tubes.watch!(tube_name)
|
90
90
|
silenced(3) { @res = connection.tubes.reserve }
|
91
|
-
return @res, @res.body
|
91
|
+
return @res, JSON.parse(@res.body)
|
92
92
|
end
|
93
93
|
|
94
94
|
# clear_jobs!('foo')
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: backburner
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.2.
|
4
|
+
version: 0.2.6
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-11-
|
12
|
+
date: 2012-11-13 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: beaneater
|
@@ -18,7 +18,7 @@ dependencies:
|
|
18
18
|
requirements:
|
19
19
|
- - ~>
|
20
20
|
- !ruby/object:Gem::Version
|
21
|
-
version: 0.
|
21
|
+
version: 0.2.0
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
24
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,7 +26,7 @@ dependencies:
|
|
26
26
|
requirements:
|
27
27
|
- - ~>
|
28
28
|
- !ruby/object:Gem::Version
|
29
|
-
version: 0.
|
29
|
+
version: 0.2.0
|
30
30
|
- !ruby/object:Gem::Dependency
|
31
31
|
name: dante
|
32
32
|
requirement: !ruby/object:Gem::Requirement
|
@@ -182,3 +182,4 @@ test_files:
|
|
182
182
|
- test/simple_worker_test.rb
|
183
183
|
- test/test_helper.rb
|
184
184
|
- test/worker_test.rb
|
185
|
+
has_rdoc:
|