bbqueue 0.0.1 → 0.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +40 -10
- data/lib/bbqueue/consumer.rb +45 -2
- data/lib/bbqueue/version.rb +1 -1
- data/test/bbqueue/consumer_test.rb +92 -9
- data/test/bbqueue/test_job.rb +2 -2
- data/test/bbqueue/test_logger.rb +22 -13
- data/test/test_helper.rb +4 -2
- metadata +2 -2
data/README.md
CHANGED
@@ -2,12 +2,13 @@
|
|
2
2
|
|
3
3
|
[](http://travis-ci.org/mrkamel/bbqueue)
|
4
4
|
[](https://codeclimate.com/github/mrkamel/bbqueue)
|
5
|
+
[](http://badge.fury.io/rb/bbqueue)
|
5
6
|
[](http://stillmaintained.com/mrkamel/bbqueue)
|
6
7
|
[](https://gemnasium.com/mrkamel/bbqueue)
|
7
8
|
|
8
9
|
BBQueue is an opinionated ruby gem to queue and process background jobs. Other
|
9
|
-
gems for this purpose usually don't work with ruby objects and serialize
|
10
|
-
|
10
|
+
gems for this purpose usually don't work with ruby objects and serialize method
|
11
|
+
arguments only. Instead, BBQueue jobs are simple ruby objects:
|
11
12
|
|
12
13
|
```ruby
|
13
14
|
MyQueue.enqueue MyJob.new
|
@@ -16,7 +17,7 @@ MyQueue.enqueue MyJob.new
|
|
16
17
|
BBQueue jobs need to fulfill the following interface:
|
17
18
|
|
18
19
|
1. The object contains an instance method `#work` without any arguments
|
19
|
-
2. The object must be serializable via `Marshal.dump` and `Marshal.load`
|
20
|
+
2. The object (instance) must be serializable via `Marshal.dump` and `Marshal.load`
|
20
21
|
|
21
22
|
## Installation
|
22
23
|
|
@@ -36,8 +37,9 @@ Or install it yourself as:
|
|
36
37
|
|
37
38
|
## Usage
|
38
39
|
|
39
|
-
BBQueue is currently built on top of the
|
40
|
-
|
40
|
+
BBQueue is currently built on top of the
|
41
|
+
[stalking](https://github.com/mrkamel/stalking) gem. Therefore, you have to
|
42
|
+
install [beanstalkd](http://kr.github.io/beanstalkd/).
|
41
43
|
|
42
44
|
### Producer
|
43
45
|
|
@@ -86,8 +88,6 @@ end
|
|
86
88
|
To process the enqueued jobs, create a file, e.g. jobs.rb:
|
87
89
|
|
88
90
|
```ruby
|
89
|
-
require "bbqueue"
|
90
|
-
|
91
91
|
BBQueue::Consumer.new "default"
|
92
92
|
```
|
93
93
|
|
@@ -96,10 +96,11 @@ and run it via:
|
|
96
96
|
$ bbqueue jobs.rb
|
97
97
|
|
98
98
|
BBQueue will loop through all the jobs, run them, and will then wait for new
|
99
|
-
ones. You can pass multiple queue names and `stalking`-specific
|
99
|
+
ones (no polling). You can pass multiple queue names and `stalking`-specific
|
100
|
+
options:
|
100
101
|
|
101
102
|
```ruby
|
102
|
-
BBQueue.new ["default", "important"], :logger => ..., :servers => ...
|
103
|
+
BBQueue::Consumer.new ["default", "important"], :logger => ..., :servers => ...
|
103
104
|
```
|
104
105
|
|
105
106
|
By using a separate file like `jobs.rb`, you can load your environment or do
|
@@ -116,6 +117,35 @@ SomeLogger.info "jobs booted into #{Rails.env.inspect} environment"
|
|
116
117
|
BBQueue::Consumer.new "background", :logger => SomeLogger
|
117
118
|
```
|
118
119
|
|
120
|
+
### Forking Consumers
|
121
|
+
|
122
|
+
By default, BBQueue does not fork a process for every job, but
|
123
|
+
it already ships with the ability to do so.
|
124
|
+
|
125
|
+
To make BBQueue fork, create a custom consumer:
|
126
|
+
|
127
|
+
```ruby
|
128
|
+
class ForkingConsumer < BBQueue::Consumer
|
129
|
+
def fork?
|
130
|
+
true
|
131
|
+
end
|
132
|
+
|
133
|
+
def before_fork
|
134
|
+
# ...
|
135
|
+
end
|
136
|
+
|
137
|
+
def after_fork
|
138
|
+
# ...
|
139
|
+
end
|
140
|
+
end
|
141
|
+
```
|
142
|
+
|
143
|
+
Then, start your custom consumer in favor of BBQueue's default one:
|
144
|
+
|
145
|
+
```ruby
|
146
|
+
ForkingConsumer.new ["queue1", "queue2", ...], ...
|
147
|
+
```
|
148
|
+
|
119
149
|
## Graceful Termination
|
120
150
|
|
121
151
|
Like for the `stalking` gem, you can stop a worker gracefully by sending a QUIT
|
@@ -123,7 +153,7 @@ signal to it. The worker will finish its current job and terminate afterwards.
|
|
123
153
|
|
124
154
|
## Contributing
|
125
155
|
|
126
|
-
1. Fork it ( https://github.com/
|
156
|
+
1. Fork it ( https://github.com/mrkamel/bbqueue/fork )
|
127
157
|
2. Create your feature branch (`git checkout -b my-new-feature`)
|
128
158
|
3. Commit your changes (`git commit -am 'Add some feature'`)
|
129
159
|
4. Push to the branch (`git push origin my-new-feature`)
|
data/lib/bbqueue/consumer.rb
CHANGED
@@ -1,13 +1,56 @@
|
|
1
1
|
|
2
2
|
module BBQueue
|
3
3
|
class Consumer
|
4
|
+
attr_accessor :logger
|
5
|
+
|
6
|
+
def before_fork
|
7
|
+
# Nothing
|
8
|
+
end
|
9
|
+
|
10
|
+
def after_fork
|
11
|
+
# Nothing
|
12
|
+
end
|
13
|
+
|
14
|
+
def fork?
|
15
|
+
false
|
16
|
+
end
|
17
|
+
|
18
|
+
def fork_and_wait
|
19
|
+
if fork?
|
20
|
+
before_fork
|
21
|
+
|
22
|
+
Process.fork do
|
23
|
+
after_fork
|
24
|
+
|
25
|
+
yield
|
26
|
+
end
|
27
|
+
|
28
|
+
Process.wait
|
29
|
+
else
|
30
|
+
yield
|
31
|
+
end
|
32
|
+
end
|
33
|
+
|
34
|
+
def work(job, queue_name)
|
35
|
+
fork_and_wait do
|
36
|
+
begin
|
37
|
+
job.work
|
38
|
+
rescue Timeout::Error, StandardError => e
|
39
|
+
logger.error "Job #{job.inspect} on #{queue_name.inspect} failed"
|
40
|
+
logger.error e
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
4
45
|
def initialize(queue_names, options = {})
|
5
|
-
logger = options[:logger] || BBQueue::NullLogger.new
|
46
|
+
self.logger = logger = options[:logger] || BBQueue::NullLogger.new
|
47
|
+
|
48
|
+
consumer = self
|
6
49
|
|
7
50
|
Stalking::Consumer.new options.merge(:logger => BBQueue::FatalLogger.new(logger)) do
|
8
51
|
Array(queue_names).each do |queue_name|
|
9
52
|
job queue_name do |args|
|
10
|
-
BBQueue::Serializer.load(args["object"])
|
53
|
+
consumer.work BBQueue::Serializer.load(args["object"]), queue_name
|
11
54
|
end
|
12
55
|
end
|
13
56
|
|
data/lib/bbqueue/version.rb
CHANGED
@@ -2,18 +2,101 @@
|
|
2
2
|
require File.expand_path("../../test_helper", __FILE__)
|
3
3
|
|
4
4
|
class BBQueue::ConsumerTest < MiniTest::Test
|
5
|
-
|
6
|
-
|
5
|
+
class ForkingConsumer < BBQueue::Consumer
|
6
|
+
attr_accessor :tempfile
|
7
7
|
|
8
|
-
|
8
|
+
def initialize(tempfile, *args)
|
9
|
+
self.tempfile = tempfile
|
10
|
+
|
11
|
+
super *args
|
12
|
+
end
|
13
|
+
|
14
|
+
def fork?
|
15
|
+
true
|
16
|
+
end
|
17
|
+
|
18
|
+
def before_fork
|
19
|
+
tempfile.puts "before_fork"
|
20
|
+
end
|
21
|
+
|
22
|
+
def after_fork
|
23
|
+
tempfile.puts "after_fork"
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
class NonForkingConsumer < ForkingConsumer
|
28
|
+
def fork?
|
29
|
+
false
|
30
|
+
end
|
31
|
+
end
|
32
|
+
|
33
|
+
class LoggingJob
|
34
|
+
attr_accessor :tempfile_path
|
35
|
+
|
36
|
+
def initialize(tempfile_path)
|
37
|
+
self.tempfile_path = tempfile_path
|
38
|
+
end
|
39
|
+
|
40
|
+
def work
|
41
|
+
open(tempfile_path, "w+") { |stream| stream.puts "work" }
|
42
|
+
end
|
43
|
+
end
|
44
|
+
|
45
|
+
class NullJob
|
46
|
+
def work
|
47
|
+
# Nothing
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
class RaisingJob
|
52
|
+
def work
|
53
|
+
raise "BLUBB"
|
54
|
+
end
|
55
|
+
end
|
56
|
+
|
57
|
+
def test_non_forking_job
|
58
|
+
consumer_tempfile = Tempfile.new("consumer")
|
59
|
+
consumer = NonForkingConsumer.new(consumer_tempfile, "default")
|
60
|
+
|
61
|
+
job_tempfile = Tempfile.new("job")
|
62
|
+
|
63
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
|
64
|
+
|
65
|
+
refute_includes File.read(consumer_tempfile.path), "before_fork"
|
66
|
+
refute_includes File.read(consumer_tempfile.path), "after_fork"
|
67
|
+
|
68
|
+
assert_includes File.read(job_tempfile.path), "work"
|
69
|
+
end
|
70
|
+
|
71
|
+
def test_forking_job
|
72
|
+
consumer_tempfile = Tempfile.new("consumer")
|
73
|
+
consumer = ForkingConsumer.new(consumer_tempfile, "default")
|
74
|
+
|
75
|
+
job_tempfile = Tempfile.new("job")
|
76
|
+
|
77
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
|
78
|
+
|
79
|
+
assert_includes File.read(consumer_tempfile.path), "before_fork"
|
80
|
+
assert_includes File.read(consumer_tempfile.path), "after_fork"
|
81
|
+
|
82
|
+
assert_includes File.read(job_tempfile.path), "work"
|
83
|
+
end
|
84
|
+
|
85
|
+
def test_forking_job_with_raise
|
86
|
+
logger = BBQueue::TestLogger.new
|
87
|
+
consumer = ForkingConsumer.new(Tempfile.new("consumer"), "default", :logger => logger)
|
88
|
+
|
89
|
+
assert_difference "logger.count /^error:/", 2 do
|
90
|
+
Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(RaisingJob.new) }
|
91
|
+
end
|
9
92
|
end
|
10
93
|
|
11
94
|
def test_error
|
12
95
|
logger = BBQueue::TestLogger.new
|
13
96
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
14
97
|
|
15
|
-
assert_difference "logger.count", 2 do
|
16
|
-
Stalking::Consumer.instances.last.run_error
|
98
|
+
assert_difference "logger.count /^error:/", 2 do
|
99
|
+
Stalking::Consumer.instances.last.run_error "Error", "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
17
100
|
end
|
18
101
|
end
|
19
102
|
|
@@ -21,8 +104,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
|
|
21
104
|
logger = BBQueue::TestLogger.new
|
22
105
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
23
106
|
|
24
|
-
assert_difference "logger.count" do
|
25
|
-
Stalking::Consumer.instances.last.run_before
|
107
|
+
assert_difference "logger.count /^info:/" do
|
108
|
+
Stalking::Consumer.instances.last.run_before "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
26
109
|
end
|
27
110
|
end
|
28
111
|
|
@@ -30,8 +113,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
|
|
30
113
|
logger = BBQueue::TestLogger.new
|
31
114
|
consumer = BBQueue::Consumer.new("default", :logger => logger)
|
32
115
|
|
33
|
-
assert_difference "logger.count" do
|
34
|
-
Stalking::Consumer.instances.last.run_after
|
116
|
+
assert_difference "logger.count /^info:/" do
|
117
|
+
Stalking::Consumer.instances.last.run_after "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
|
35
118
|
end
|
36
119
|
end
|
37
120
|
end
|
data/test/bbqueue/test_job.rb
CHANGED
data/test/bbqueue/test_logger.rb
CHANGED
@@ -1,31 +1,40 @@
|
|
1
1
|
|
2
|
+
require "logger"
|
3
|
+
|
2
4
|
class BBQueue::TestLogger
|
5
|
+
attr_accessor :tempfile, :logger
|
6
|
+
|
3
7
|
def initialize
|
4
|
-
|
8
|
+
self.tempfile = Tempfile.new("logger")
|
9
|
+
|
10
|
+
logger = Logger.new(tempfile.path)
|
11
|
+
logger.formatter = proc { |severity, datetime, progname, msg| "#{severity.downcase}: #{msg}\n" }
|
12
|
+
|
13
|
+
self.logger = logger
|
5
14
|
end
|
6
15
|
|
7
|
-
def debug(
|
8
|
-
|
16
|
+
def debug(*args)
|
17
|
+
logger.debug(*args)
|
9
18
|
end
|
10
19
|
|
11
|
-
def info(
|
12
|
-
|
20
|
+
def info(*args)
|
21
|
+
logger.info(*args)
|
13
22
|
end
|
14
23
|
|
15
|
-
def warn(
|
16
|
-
|
24
|
+
def warn(*args)
|
25
|
+
logger.warn(*args)
|
17
26
|
end
|
18
27
|
|
19
|
-
def error(
|
20
|
-
|
28
|
+
def error(*args)
|
29
|
+
logger.error(*args)
|
21
30
|
end
|
22
31
|
|
23
|
-
def fatal(
|
24
|
-
|
32
|
+
def fatal(*args)
|
33
|
+
logger.fatal(*args)
|
25
34
|
end
|
26
35
|
|
27
|
-
def count
|
28
|
-
|
36
|
+
def count(regex = /.*/)
|
37
|
+
File.read(tempfile.path).lines.grep(regex).count
|
29
38
|
end
|
30
39
|
end
|
31
40
|
|
data/test/test_helper.rb
CHANGED
@@ -2,6 +2,7 @@
|
|
2
2
|
require "bbqueue"
|
3
3
|
require "minitest"
|
4
4
|
require "minitest/autorun"
|
5
|
+
require "tempfile"
|
5
6
|
|
6
7
|
require File.expand_path("../bbqueue/stalking", __FILE__)
|
7
8
|
require File.expand_path("../bbqueue/test_job", __FILE__)
|
@@ -17,9 +18,10 @@ class MiniTest::Test
|
|
17
18
|
def assert_difference(expression, difference = 1, message = nil, &block)
|
18
19
|
expressions = Array(expression)
|
19
20
|
|
20
|
-
exps = expressions.map
|
21
|
+
exps = expressions.map do |e|
|
21
22
|
e.respond_to?(:call) ? e : lambda { eval(e, block.binding) }
|
22
|
-
|
23
|
+
end
|
24
|
+
|
23
25
|
before = exps.map { |e| e.call }
|
24
26
|
|
25
27
|
yield
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: bbqueue
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2015-07-02 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: bundler
|