bbqueue 0.0.1 → 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
data/README.md CHANGED
@@ -2,12 +2,13 @@
2
2
 
3
3
  [![Build Status](https://secure.travis-ci.org/mrkamel/bbqueue.png?branch=master)](http://travis-ci.org/mrkamel/bbqueue)
4
4
  [![Code Quality](https://codeclimate.com/github/mrkamel/bbqueue.png)](https://codeclimate.com/github/mrkamel/bbqueue)
5
+ [![Gem Version](https://badge.fury.io/rb/bbqueue.svg)](http://badge.fury.io/rb/bbqueue)
5
6
  [![Still Maintained](http://stillmaintained.com/mrkamel/bbqueue.png)](http://stillmaintained.com/mrkamel/bbqueue)
6
7
  [![Dependency Status](https://gemnasium.com/mrkamel/bbqueue.png?travis)](https://gemnasium.com/mrkamel/bbqueue)
7
8
 
8
9
  BBQueue is an opinionated ruby gem to queue and process background jobs. Other
9
- gems for this purpose usually don't work with ruby objects and serialize only
10
- method arguments. Instead, BBQueue jobs are simple ruby objects:
10
+ gems for this purpose usually don't work with ruby objects and serialize method
11
+ arguments only. Instead, BBQueue jobs are simple ruby objects:
11
12
 
12
13
  ```ruby
13
14
  MyQueue.enqueue MyJob.new
@@ -16,7 +17,7 @@ MyQueue.enqueue MyJob.new
16
17
  BBQueue jobs need to fulfill the following interface:
17
18
 
18
19
  1. The object contains an instance method `#work` without any arguments
19
- 2. The object must be serializable via `Marshal.dump` and `Marshal.load`
20
+ 2. The object (instance) must be serializable via `Marshal.dump` and `Marshal.load`
20
21
 
21
22
  ## Installation
22
23
 
@@ -36,8 +37,9 @@ Or install it yourself as:
36
37
 
37
38
  ## Usage
38
39
 
39
- BBQueue is currently built on top of the `stalking` gem. Therefore, you have to
40
- install beanstalkd.
40
+ BBQueue is currently built on top of the
41
+ [stalking](https://github.com/mrkamel/stalking) gem. Therefore, you have to
42
+ install [beanstalkd](http://kr.github.io/beanstalkd/).
41
43
 
42
44
  ### Producer
43
45
 
@@ -86,8 +88,6 @@ end
86
88
  To process the enqueued jobs, create a file, e.g. jobs.rb:
87
89
 
88
90
  ```ruby
89
- require "bbqueue"
90
-
91
91
  BBQueue::Consumer.new "default"
92
92
  ```
93
93
 
@@ -96,10 +96,11 @@ and run it via:
96
96
  $ bbqueue jobs.rb
97
97
 
98
98
  BBQueue will loop through all the jobs, run them, and will then wait for new
99
- ones. You can pass multiple queue names and `stalking`-specific options:
99
+ ones (no polling). You can pass multiple queue names and `stalking`-specific
100
+ options:
100
101
 
101
102
  ```ruby
102
- BBQueue.new ["default", "important"], :logger => ..., :servers => ...
103
+ BBQueue::Consumer.new ["default", "important"], :logger => ..., :servers => ...
103
104
  ```
104
105
 
105
106
  By using a separate file like `jobs.rb`, you can load your environment or do
@@ -116,6 +117,35 @@ SomeLogger.info "jobs booted into #{Rails.env.inspect} environment"
116
117
  BBQueue::Consumer.new "background", :logger => SomeLogger
117
118
  ```
118
119
 
120
+ ### Forking Consumers
121
+
122
+ By default, BBQueue does not fork a process for every job, but
123
+ it already ships with the ability to do so.
124
+
125
+ To make BBQueue fork, create a custom consumer:
126
+
127
+ ```ruby
128
+ class ForkingConsumer < BBQueue::Consumer
129
+ def fork?
130
+ true
131
+ end
132
+
133
+ def before_fork
134
+ # ...
135
+ end
136
+
137
+ def after_fork
138
+ # ...
139
+ end
140
+ end
141
+ ```
142
+
143
+ Then, start your custom consumer in favor of BBQueue's default one:
144
+
145
+ ```ruby
146
+ ForkingConsumer.new ["queue1", "queue2", ...], ...
147
+ ```
148
+
119
149
  ## Graceful Termination
120
150
 
121
151
  Like for the `stalking` gem, you can stop a worker gracefully by sending a QUIT
@@ -123,7 +153,7 @@ signal to it. The worker will finish its current job and terminate afterwards.
123
153
 
124
154
  ## Contributing
125
155
 
126
- 1. Fork it ( https://github.com/[my-github-username]/bbqueue/fork )
156
+ 1. Fork it ( https://github.com/mrkamel/bbqueue/fork )
127
157
  2. Create your feature branch (`git checkout -b my-new-feature`)
128
158
  3. Commit your changes (`git commit -am 'Add some feature'`)
129
159
  4. Push to the branch (`git push origin my-new-feature`)
@@ -1,13 +1,56 @@
1
1
 
2
2
  module BBQueue
3
3
  class Consumer
4
+ attr_accessor :logger
5
+
6
+ def before_fork
7
+ # Nothing
8
+ end
9
+
10
+ def after_fork
11
+ # Nothing
12
+ end
13
+
14
+ def fork?
15
+ false
16
+ end
17
+
18
+ def fork_and_wait
19
+ if fork?
20
+ before_fork
21
+
22
+ Process.fork do
23
+ after_fork
24
+
25
+ yield
26
+ end
27
+
28
+ Process.wait
29
+ else
30
+ yield
31
+ end
32
+ end
33
+
34
+ def work(job, queue_name)
35
+ fork_and_wait do
36
+ begin
37
+ job.work
38
+ rescue Timeout::Error, StandardError => e
39
+ logger.error "Job #{job.inspect} on #{queue_name.inspect} failed"
40
+ logger.error e
41
+ end
42
+ end
43
+ end
44
+
4
45
  def initialize(queue_names, options = {})
5
- logger = options[:logger] || BBQueue::NullLogger.new
46
+ self.logger = logger = options[:logger] || BBQueue::NullLogger.new
47
+
48
+ consumer = self
6
49
 
7
50
  Stalking::Consumer.new options.merge(:logger => BBQueue::FatalLogger.new(logger)) do
8
51
  Array(queue_names).each do |queue_name|
9
52
  job queue_name do |args|
10
- BBQueue::Serializer.load(args["object"]).work
53
+ consumer.work BBQueue::Serializer.load(args["object"]), queue_name
11
54
  end
12
55
  end
13
56
 
@@ -1,3 +1,3 @@
1
1
  module BBQueue
2
- VERSION = "0.0.1"
2
+ VERSION = "0.0.2"
3
3
  end
@@ -2,18 +2,101 @@
2
2
  require File.expand_path("../../test_helper", __FILE__)
3
3
 
4
4
  class BBQueue::ConsumerTest < MiniTest::Test
5
- def test_job
6
- consumer = BBQueue::Consumer.new("default")
5
+ class ForkingConsumer < BBQueue::Consumer
6
+ attr_accessor :tempfile
7
7
 
8
- assert_equal 3, Stalking::Consumer.instances.last.run_job("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
8
+ def initialize(tempfile, *args)
9
+ self.tempfile = tempfile
10
+
11
+ super *args
12
+ end
13
+
14
+ def fork?
15
+ true
16
+ end
17
+
18
+ def before_fork
19
+ tempfile.puts "before_fork"
20
+ end
21
+
22
+ def after_fork
23
+ tempfile.puts "after_fork"
24
+ end
25
+ end
26
+
27
+ class NonForkingConsumer < ForkingConsumer
28
+ def fork?
29
+ false
30
+ end
31
+ end
32
+
33
+ class LoggingJob
34
+ attr_accessor :tempfile_path
35
+
36
+ def initialize(tempfile_path)
37
+ self.tempfile_path = tempfile_path
38
+ end
39
+
40
+ def work
41
+ open(tempfile_path, "w+") { |stream| stream.puts "work" }
42
+ end
43
+ end
44
+
45
+ class NullJob
46
+ def work
47
+ # Nothing
48
+ end
49
+ end
50
+
51
+ class RaisingJob
52
+ def work
53
+ raise "BLUBB"
54
+ end
55
+ end
56
+
57
+ def test_non_forking_job
58
+ consumer_tempfile = Tempfile.new("consumer")
59
+ consumer = NonForkingConsumer.new(consumer_tempfile, "default")
60
+
61
+ job_tempfile = Tempfile.new("job")
62
+
63
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
64
+
65
+ refute_includes File.read(consumer_tempfile.path), "before_fork"
66
+ refute_includes File.read(consumer_tempfile.path), "after_fork"
67
+
68
+ assert_includes File.read(job_tempfile.path), "work"
69
+ end
70
+
71
+ def test_forking_job
72
+ consumer_tempfile = Tempfile.new("consumer")
73
+ consumer = ForkingConsumer.new(consumer_tempfile, "default")
74
+
75
+ job_tempfile = Tempfile.new("job")
76
+
77
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
78
+
79
+ assert_includes File.read(consumer_tempfile.path), "before_fork"
80
+ assert_includes File.read(consumer_tempfile.path), "after_fork"
81
+
82
+ assert_includes File.read(job_tempfile.path), "work"
83
+ end
84
+
85
+ def test_forking_job_with_raise
86
+ logger = BBQueue::TestLogger.new
87
+ consumer = ForkingConsumer.new(Tempfile.new("consumer"), "default", :logger => logger)
88
+
89
+ assert_difference "logger.count /^error:/", 2 do
90
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(RaisingJob.new) }
91
+ end
9
92
  end
10
93
 
11
94
  def test_error
12
95
  logger = BBQueue::TestLogger.new
13
96
  consumer = BBQueue::Consumer.new("default", :logger => logger)
14
97
 
15
- assert_difference "logger.count", 2 do
16
- Stalking::Consumer.instances.last.run_error("Error", "default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
98
+ assert_difference "logger.count /^error:/", 2 do
99
+ Stalking::Consumer.instances.last.run_error "Error", "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
17
100
  end
18
101
  end
19
102
 
@@ -21,8 +104,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
21
104
  logger = BBQueue::TestLogger.new
22
105
  consumer = BBQueue::Consumer.new("default", :logger => logger)
23
106
 
24
- assert_difference "logger.count" do
25
- Stalking::Consumer.instances.last.run_before("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
107
+ assert_difference "logger.count /^info:/" do
108
+ Stalking::Consumer.instances.last.run_before "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
26
109
  end
27
110
  end
28
111
 
@@ -30,8 +113,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
30
113
  logger = BBQueue::TestLogger.new
31
114
  consumer = BBQueue::Consumer.new("default", :logger => logger)
32
115
 
33
- assert_difference "logger.count" do
34
- Stalking::Consumer.instances.last.run_after("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
116
+ assert_difference "logger.count /^info:/" do
117
+ Stalking::Consumer.instances.last.run_after "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
35
118
  end
36
119
  end
37
120
  end
@@ -1,6 +1,6 @@
1
1
 
2
2
  class BBQueue::TestJob
3
- attr_accessor :x, :y
3
+ attr_accessor :x, :y, :z
4
4
 
5
5
  def initialize(x, y)
6
6
  self.x = x
@@ -12,7 +12,7 @@ class BBQueue::TestJob
12
12
  end
13
13
 
14
14
  def work
15
- x + y
15
+ self.z = x + y
16
16
  end
17
17
  end
18
18
 
@@ -1,31 +1,40 @@
1
1
 
2
+ require "logger"
3
+
2
4
  class BBQueue::TestLogger
5
+ attr_accessor :tempfile, :logger
6
+
3
7
  def initialize
4
- @messages = []
8
+ self.tempfile = Tempfile.new("logger")
9
+
10
+ logger = Logger.new(tempfile.path)
11
+ logger.formatter = proc { |severity, datetime, progname, msg| "#{severity.downcase}: #{msg}\n" }
12
+
13
+ self.logger = logger
5
14
  end
6
15
 
7
- def debug(object)
8
- @messages.push object
16
+ def debug(*args)
17
+ logger.debug(*args)
9
18
  end
10
19
 
11
- def info(object)
12
- @messages.push object
20
+ def info(*args)
21
+ logger.info(*args)
13
22
  end
14
23
 
15
- def warn(object)
16
- @messages.push object
24
+ def warn(*args)
25
+ logger.warn(*args)
17
26
  end
18
27
 
19
- def error(object)
20
- @messages.push object
28
+ def error(*args)
29
+ logger.error(*args)
21
30
  end
22
31
 
23
- def fatal(object)
24
- @messages.push object
32
+ def fatal(*args)
33
+ logger.fatal(*args)
25
34
  end
26
35
 
27
- def count
28
- @messages.size
36
+ def count(regex = /.*/)
37
+ File.read(tempfile.path).lines.grep(regex).count
29
38
  end
30
39
  end
31
40
 
@@ -2,6 +2,7 @@
2
2
  require "bbqueue"
3
3
  require "minitest"
4
4
  require "minitest/autorun"
5
+ require "tempfile"
5
6
 
6
7
  require File.expand_path("../bbqueue/stalking", __FILE__)
7
8
  require File.expand_path("../bbqueue/test_job", __FILE__)
@@ -17,9 +18,10 @@ class MiniTest::Test
17
18
  def assert_difference(expression, difference = 1, message = nil, &block)
18
19
  expressions = Array(expression)
19
20
 
20
- exps = expressions.map { |e|
21
+ exps = expressions.map do |e|
21
22
  e.respond_to?(:call) ? e : lambda { eval(e, block.binding) }
22
- }
23
+ end
24
+
23
25
  before = exps.map { |e| e.call }
24
26
 
25
27
  yield
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: bbqueue
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2014-08-06 00:00:00.000000000 Z
12
+ date: 2015-07-02 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: bundler