bbqueue 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +40 -10
- data/lib/bbqueue/consumer.rb +45 -2
- data/lib/bbqueue/version.rb +1 -1
- data/test/bbqueue/consumer_test.rb +92 -9
- data/test/bbqueue/test_job.rb +2 -2
- data/test/bbqueue/test_logger.rb +22 -13
- data/test/test_helper.rb +4 -2
- metadata +2 -2
data/README.md
CHANGED
@@ -2,12 +2,13 @@
|
|
2
2
|
|
3
3
|
[![Build Status](https://secure.travis-ci.org/mrkamel/bbqueue.png?branch=master)](http://travis-ci.org/mrkamel/bbqueue)
|
4
4
|
[![Code Quality](https://codeclimate.com/github/mrkamel/bbqueue.png)](https://codeclimate.com/github/mrkamel/bbqueue)
|
5
|
+
[![Gem Version](https://badge.fury.io/rb/bbqueue.svg)](http://badge.fury.io/rb/bbqueue)
|
5
6
|
[![Still Maintained](http://stillmaintained.com/mrkamel/bbqueue.png)](http://stillmaintained.com/mrkamel/bbqueue)
|
6
7
|
[![Dependency Status](https://gemnasium.com/mrkamel/bbqueue.png?travis)](https://gemnasium.com/mrkamel/bbqueue)
|
7
8
|
|
8
9
|
BBQueue is an opinionated ruby gem to queue and process background jobs. Other
|
9
|
-
gems for this purpose usually don't work with ruby objects and serialize
|
10
|
-
|
10
|
+
gems for this purpose usually don't work with ruby objects and serialize method
|
11
|
+
arguments only. Instead, BBQueue jobs are simple ruby objects:
|
11
12
|
|
12
13
|
```ruby
|
13
14
|
MyQueue.enqueue MyJob.new
|
@@ -16,7 +17,7 @@ MyQueue.enqueue MyJob.new
|
|
16
17
|
BBQueue jobs need to fulfill the following interface:
|
17
18
|
|
18
19
|
1. The object contains an instance method `#work` without any arguments
|
19
|
-
2. The object must be serializable via `Marshal.dump` and `Marshal.load`
|
20
|
+
2. The object (instance) must be serializable via `Marshal.dump` and `Marshal.load`
|
20
21
|
|
21
22
|
## Installation
|
22
23
|
|
@@ -36,8 +37,9 @@ Or install it yourself as:
|
|
36
37
|
|
37
38
|
## Usage
|
38
39
|
|
39
|
-
BBQueue is currently built on top of the
|
40
|
-
|
40
|
+
BBQueue is currently built on top of the
|
41
|
+
[stalking](https://github.com/mrkamel/stalking) gem. Therefore, you have to
|
42
|
+
install [beanstalkd](http://kr.github.io/beanstalkd/).
|
41
43
|
|
42
44
|
### Producer
|
43
45
|
|
@@ -86,8 +88,6 @@ end
|
|
86
88
|
To process the enqueued jobs, create a file, e.g. jobs.rb:
|
87
89
|
|
88
90
|
```ruby
|
89
|
-
require "bbqueue"
|
90
|
-
|
91
91
|
BBQueue::Consumer.new "default"
|
92
92
|
```
|
93
93
|
|
@@ -96,10 +96,11 @@ and run it via:
|
|
96
96
|
$ bbqueue jobs.rb
|
97
97
|
|
98
98
|
BBQueue will loop through all the jobs, run them, and will then wait for new
|
99
|
-
ones. You can pass multiple queue names and `stalking`-specific
|
99
|
+
ones (no polling). You can pass multiple queue names and `stalking`-specific
|
100
|
+
options:
|
100
101
|
|
101
102
|
```ruby
|
102
|
-
BBQueue.new ["default", "important"], :logger => ..., :servers => ...
|
103
|
+
BBQueue::Consumer.new ["default", "important"], :logger => ..., :servers => ...
|
103
104
|
```
|
104
105
|
|
105
106
|
By using a separate file like `jobs.rb`, you can load your environment or do
|
@@ -116,6 +117,35 @@ SomeLogger.info "jobs booted into #{Rails.env.inspect} environment"
|
|
116
117
|
BBQueue::Consumer.new "background", :logger => SomeLogger
|
117
118
|
```
|
118
119
|
|
120
|
+
### Forking Consumers
|
121
|
+
|
122
|
+
By default, BBQueue does not fork a process for every job, but
|
123
|
+
it already ships with the ability to do so.
|
124
|
+
|
125
|
+
To make BBQueue fork, create a custom consumer:
|
126
|
+
|
127
|
+
```ruby
|
128
|
+
class ForkingConsumer < BBQueue::Consumer
|
129
|
+
def fork?
|
130
|
+
true
|
131
|
+
end
|
132
|
+
|
133
|
+
def before_fork
|
134
|
+
# ...
|
135
|
+
end
|
136
|
+
|
137
|
+
def after_fork
|
138
|
+
# ...
|
139
|
+
end
|
140
|
+
end
|
141
|
+
```
|
142
|
+
|
143
|
+
Then, start your custom consumer in favor of BBQueue's default one:
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
ForkingConsumer.new ["queue1", "queue2", ...], ...
|
147
|
+
```
|
148
|
+
|
119
149
|
## Graceful Termination
|
120
150
|
|
121
151
|
Like for the `stalking` gem, you can stop a worker gracefully by sending a QUIT
|
@@ -123,7 +153,7 @@ signal to it. The worker will finish its current job and terminate afterwards.
|
|
123
153
|
|
124
154
|
## Contributing
|
125
155
|
|
126
|
-
1. Fork it ( https://github.com/
|
156
|
+
1. Fork it ( https://github.com/mrkamel/bbqueue/fork )
|
127
157
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
128
158
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
129
159
|
4. Push to the branch (`git push origin my-new-feature`)
|
data/lib/bbqueue/consumer.rb
CHANGED
@@ -1,13 +1,56 @@
|
|
1
1
|
|
2
2
|
module BBQueue
|
3
3
|
class Consumer
|
4
|
+
attr_accessor :logger
|
5
|
+
|
6
|
+
def before_fork
|
7
|
+
# Nothing
|
8
|
+
end
|
9
|
+
|
10
|
+
def after_fork
|
11
|
+
# Nothing
|
12
|
+
end
|
13
|
+
|
14
|
+
def fork?
|
15
|
+
false
|
16
|
+
end
|
17
|
+
|
18
|
+
def fork_and_wait
|
19
|
+
if fork?
|
20
|
+
before_fork
|
21
|
+
|
22
|
+
Process.fork do
|
23
|
+
after_fork
|
24
|
+
|
25
|
+
yield
|
26
|
+
end
|
27
|
+
|
28
|
+
Process.wait
|
29
|
+
else
|
30
|
+
yield
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def work(job, queue_name)
|
35
|
+
fork_and_wait do
|
36
|
+
begin
|
37
|
+
job.work
|
38
|
+
rescue Timeout::Error, StandardError => e
|
39
|
+
logger.error "Job #{job.inspect} on #{queue_name.inspect} failed"
|
40
|
+
logger.error e
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
4
45
|
def initialize(queue_names, options = {})
|
5
|
-
logger = options[:logger] || BBQueue::NullLogger.new
|
46
|
+
self.logger = logger = options[:logger] || BBQueue::NullLogger.new
|
47
|
+
|
48
|
+
consumer = self
|
6
49
|
|
7
50
|
Stalking::Consumer.new options.merge(:logger => BBQueue::FatalLogger.new(logger)) do
|
8
51
|
Array(queue_names).each do |queue_name|
|
9
52
|
job queue_name do |args|
|
10
|
-
BBQueue::Serializer.load(args["object"])
|
53
|
+
consumer.work BBQueue::Serializer.load(args["object"]), queue_name
|
11
54
|
end
|
12
55
|
end
|
13
56
|
|
data/lib/bbqueue/version.rb
CHANGED
@@ -2,18 +2,101 @@
|
|
2
2
|
require File.expand_path("../../test_helper", __FILE__)
|
3
3
|
|
4
4
|
class BBQueue::ConsumerTest < MiniTest::Test
|
5
|
-
|
6
|
-
|
5
|
+
class ForkingConsumer < BBQueue::Consumer
|
6
|
+
attr_accessor :tempfile
|
7
7
|
|
8
|
-
|
8
|
+
def initialize(tempfile, *args)
|
9
|
+
self.tempfile = tempfile
|
10
|
+
|
11
|
+
super *args
|
12
|
+
end
|
13
|
+
|
14
|
+
def fork?
|
15
|
+
true
|
16
|
+
end
|
17
|
+
|
18
|
+
def before_fork
|
19
|
+
tempfile.puts "before_fork"
|
20
|
+
end
|
21
|
+
|
22
|
+
def after_fork
|
23
|
+
tempfile.puts "after_fork"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
class NonForkingConsumer < ForkingConsumer
|
28
|
+
def fork?
|
29
|
+
false
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
class LoggingJob
|
34
|
+
attr_accessor :tempfile_path
|
35
|
+
|
36
|
+
def initialize(tempfile_path)
|
37
|
+
self.tempfile_path = tempfile_path
|
38
|
+
end
|
39
|
+
|
40
|
+
def work
|
41
|
+
open(tempfile_path, "w+") { |stream| stream.puts "work" }
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class NullJob
|
46
|
+
def work
|
47
|
+
# Nothing
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
class RaisingJob
|
52
|
+
def work
|
53
|
+
raise "BLUBB"
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def test_non_forking_job
|
58
|
+
consumer_tempfile = Tempfile.new("consumer")
|
59
|
+
consumer = NonForkingConsumer.new(consumer_tempfile, "default")
|
60
|
+
|
61
|
+
job_tempfile = Tempfile.new("job")
|
62
|
+
|
63
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
|
64
|
+
|
65
|
+
refute_includes File.read(consumer_tempfile.path), "before_fork"
|
66
|
+
refute_includes File.read(consumer_tempfile.path), "after_fork"
|
67
|
+
|
68
|
+
assert_includes File.read(job_tempfile.path), "work"
|
69
|
+
end
|
70
|
+
|
71
|
+
def test_forking_job
|
72
|
+
consumer_tempfile = Tempfile.new("consumer")
|
73
|
+
consumer = ForkingConsumer.new(consumer_tempfile, "default")
|
74
|
+
|
75
|
+
job_tempfile = Tempfile.new("job")
|
76
|
+
|
77
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
|
78
|
+
|
79
|
+
assert_includes File.read(consumer_tempfile.path), "before_fork"
|
80
|
+
assert_includes File.read(consumer_tempfile.path), "after_fork"
|
81
|
+
|
82
|
+
assert_includes File.read(job_tempfile.path), "work"
|
83
|
+
end
|
84
|
+
|
85
|
+
def test_forking_job_with_raise
|
86
|
+
logger = BBQueue::TestLogger.new
|
87
|
+
consumer = ForkingConsumer.new(Tempfile.new("consumer"), "default", :logger => logger)
|
88
|
+
|
89
|
+
assert_difference "logger.count /^error:/", 2 do
|
90
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(RaisingJob.new) }
|
91
|
+
end
|
9
92
|
end
|
10
93
|
|
11
94
|
def test_error
|
12
95
|
logger = BBQueue::TestLogger.new
|
13
96
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
14
97
|
|
15
|
-
assert_difference "logger.count", 2 do
|
16
|
-
Stalking::Consumer.instances.last.run_error
|
98
|
+
assert_difference "logger.count /^error:/", 2 do
|
99
|
+
Stalking::Consumer.instances.last.run_error "Error", "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
17
100
|
end
|
18
101
|
end
|
19
102
|
|
@@ -21,8 +104,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
|
|
21
104
|
logger = BBQueue::TestLogger.new
|
22
105
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
23
106
|
|
24
|
-
assert_difference "logger.count" do
|
25
|
-
Stalking::Consumer.instances.last.run_before
|
107
|
+
assert_difference "logger.count /^info:/" do
|
108
|
+
Stalking::Consumer.instances.last.run_before "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
26
109
|
end
|
27
110
|
end
|
28
111
|
|
@@ -30,8 +113,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
|
|
30
113
|
logger = BBQueue::TestLogger.new
|
31
114
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
32
115
|
|
33
|
-
assert_difference "logger.count" do
|
34
|
-
Stalking::Consumer.instances.last.run_after
|
116
|
+
assert_difference "logger.count /^info:/" do
|
117
|
+
Stalking::Consumer.instances.last.run_after "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
35
118
|
end
|
36
119
|
end
|
37
120
|
end
|
data/test/bbqueue/test_job.rb
CHANGED
data/test/bbqueue/test_logger.rb
CHANGED
@@ -1,31 +1,40 @@
|
|
1
1
|
|
2
|
+
require "logger"
|
3
|
+
|
2
4
|
class BBQueue::TestLogger
|
5
|
+
attr_accessor :tempfile, :logger
|
6
|
+
|
3
7
|
def initialize
|
4
|
-
|
8
|
+
self.tempfile = Tempfile.new("logger")
|
9
|
+
|
10
|
+
logger = Logger.new(tempfile.path)
|
11
|
+
logger.formatter = proc { |severity, datetime, progname, msg| "#{severity.downcase}: #{msg}\n" }
|
12
|
+
|
13
|
+
self.logger = logger
|
5
14
|
end
|
6
15
|
|
7
|
-
def debug(
|
8
|
-
|
16
|
+
def debug(*args)
|
17
|
+
logger.debug(*args)
|
9
18
|
end
|
10
19
|
|
11
|
-
def info(
|
12
|
-
|
20
|
+
def info(*args)
|
21
|
+
logger.info(*args)
|
13
22
|
end
|
14
23
|
|
15
|
-
def warn(
|
16
|
-
|
24
|
+
def warn(*args)
|
25
|
+
logger.warn(*args)
|
17
26
|
end
|
18
27
|
|
19
|
-
def error(
|
20
|
-
|
28
|
+
def error(*args)
|
29
|
+
logger.error(*args)
|
21
30
|
end
|
22
31
|
|
23
|
-
def fatal(
|
24
|
-
|
32
|
+
def fatal(*args)
|
33
|
+
logger.fatal(*args)
|
25
34
|
end
|
26
35
|
|
27
|
-
def count
|
28
|
-
|
36
|
+
def count(regex = /.*/)
|
37
|
+
File.read(tempfile.path).lines.grep(regex).count
|
29
38
|
end
|
30
39
|
end
|
31
40
|
|
data/test/test_helper.rb
CHANGED
@@ -2,6 +2,7 @@
|
|
2
2
|
require "bbqueue"
|
3
3
|
require "minitest"
|
4
4
|
require "minitest/autorun"
|
5
|
+
require "tempfile"
|
5
6
|
|
6
7
|
require File.expand_path("../bbqueue/stalking", __FILE__)
|
7
8
|
require File.expand_path("../bbqueue/test_job", __FILE__)
|
@@ -17,9 +18,10 @@ class MiniTest::Test
|
|
17
18
|
def assert_difference(expression, difference = 1, message = nil, &block)
|
18
19
|
expressions = Array(expression)
|
19
20
|
|
20
|
-
exps = expressions.map
|
21
|
+
exps = expressions.map do |e|
|
21
22
|
e.respond_to?(:call) ? e : lambda { eval(e, block.binding) }
|
22
|
-
|
23
|
+
end
|
24
|
+
|
23
25
|
before = exps.map { |e| e.call }
|
24
26
|
|
25
27
|
yield
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: bbqueue
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2015-07-02 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: bundler
|