bbqueue 0.0.1 → 0.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/README.md CHANGED
@@ -2,12 +2,13 @@
2
2
 
3
3
  [![Build Status](https://secure.travis-ci.org/mrkamel/bbqueue.png?branch=master)](http://travis-ci.org/mrkamel/bbqueue)
4
4
  [![Code Quality](https://codeclimate.com/github/mrkamel/bbqueue.png)](https://codeclimate.com/github/mrkamel/bbqueue)
5
+ [![Gem Version](https://badge.fury.io/rb/bbqueue.svg)](http://badge.fury.io/rb/bbqueue)
5
6
  [![Still Maintained](http://stillmaintained.com/mrkamel/bbqueue.png)](http://stillmaintained.com/mrkamel/bbqueue)
6
7
  [![Dependency Status](https://gemnasium.com/mrkamel/bbqueue.png?travis)](https://gemnasium.com/mrkamel/bbqueue)
7
8
 
8
9
  BBQueue is an opinionated ruby gem to queue and process background jobs. Other
9
- gems for this purpose usually don't work with ruby objects and serialize only
10
- method arguments. Instead, BBQueue jobs are simple ruby objects:
10
+ gems for this purpose usually don't work with ruby objects and serialize method
11
+ arguments only. Instead, BBQueue jobs are simple ruby objects:
11
12
 
12
13
  ```ruby
13
14
  MyQueue.enqueue MyJob.new
@@ -16,7 +17,7 @@ MyQueue.enqueue MyJob.new
16
17
  BBQueue jobs need to fulfill the following interface:
17
18
 
18
19
  1. The object contains an instance method `#work` without any arguments
19
- 2. The object must be serializable via `Marshal.dump` and `Marshal.load`
20
+ 2. The object (instance) must be serializable via `Marshal.dump` and `Marshal.load`
20
21
 
21
22
  ## Installation
22
23
 
@@ -36,8 +37,9 @@ Or install it yourself as:
36
37
 
37
38
  ## Usage
38
39
 
39
- BBQueue is currently built on top of the `stalking` gem. Therefore, you have to
40
- install beanstalkd.
40
+ BBQueue is currently built on top of the
41
+ [stalking](https://github.com/mrkamel/stalking) gem. Therefore, you have to
42
+ install [beanstalkd](http://kr.github.io/beanstalkd/).
41
43
 
42
44
  ### Producer
43
45
 
@@ -86,8 +88,6 @@ end
86
88
  To process the enqueued jobs, create a file, e.g. jobs.rb:
87
89
 
88
90
  ```ruby
89
- require "bbqueue"
90
-
91
91
  BBQueue::Consumer.new "default"
92
92
  ```
93
93
 
@@ -96,10 +96,11 @@ and run it via:
96
96
  $ bbqueue jobs.rb
97
97
 
98
98
  BBQueue will loop through all the jobs, run them, and will then wait for new
99
- ones. You can pass multiple queue names and `stalking`-specific options:
99
+ ones (no polling). You can pass multiple queue names and `stalking`-specific
100
+ options:
100
101
 
101
102
  ```ruby
102
- BBQueue.new ["default", "important"], :logger => ..., :servers => ...
103
+ BBQueue::Consumer.new ["default", "important"], :logger => ..., :servers => ...
103
104
  ```
104
105
 
105
106
  By using a separate file like `jobs.rb`, you can load your environment or do
@@ -116,6 +117,35 @@ SomeLogger.info "jobs booted into #{Rails.env.inspect} environment"
116
117
  BBQueue::Consumer.new "background", :logger => SomeLogger
117
118
  ```
118
119
 
120
+ ### Forking Consumers
121
+
122
+ By default, BBQueue does not fork a process for every job, but
123
+ it already ships with the ability to do so.
124
+
125
+ To make BBQueue fork, create a custom consumer:
126
+
127
+ ```ruby
128
+ class ForkingConsumer < BBQueue::Consumer
129
+ def fork?
130
+ true
131
+ end
132
+
133
+ def before_fork
134
+ # ...
135
+ end
136
+
137
+ def after_fork
138
+ # ...
139
+ end
140
+ end
141
+ ```
142
+
143
+ Then, start your custom consumer in favor of BBQueue's default one:
144
+
145
+ ```ruby
146
+ ForkingConsumer.new ["queue1", "queue2", ...], ...
147
+ ```
148
+
119
149
  ## Graceful Termination
120
150
 
121
151
  Like for the `stalking` gem, you can stop a worker gracefully by sending a QUIT
@@ -123,7 +153,7 @@ signal to it. The worker will finish its current job and terminate afterwards.
123
153
 
124
154
  ## Contributing
125
155
 
126
- 1. Fork it ( https://github.com/[my-github-username]/bbqueue/fork )
156
+ 1. Fork it ( https://github.com/mrkamel/bbqueue/fork )
127
157
  2. Create your feature branch (`git checkout -b my-new-feature`)
128
158
  3. Commit your changes (`git commit -am 'Add some feature'`)
129
159
  4. Push to the branch (`git push origin my-new-feature`)
@@ -1,13 +1,56 @@
1
1
 
2
2
  module BBQueue
3
3
  class Consumer
4
+ attr_accessor :logger
5
+
6
+ def before_fork
7
+ # Nothing
8
+ end
9
+
10
+ def after_fork
11
+ # Nothing
12
+ end
13
+
14
+ def fork?
15
+ false
16
+ end
17
+
18
+ def fork_and_wait
19
+ if fork?
20
+ before_fork
21
+
22
+ Process.fork do
23
+ after_fork
24
+
25
+ yield
26
+ end
27
+
28
+ Process.wait
29
+ else
30
+ yield
31
+ end
32
+ end
33
+
34
+ def work(job, queue_name)
35
+ fork_and_wait do
36
+ begin
37
+ job.work
38
+ rescue Timeout::Error, StandardError => e
39
+ logger.error "Job #{job.inspect} on #{queue_name.inspect} failed"
40
+ logger.error e
41
+ end
42
+ end
43
+ end
44
+
4
45
  def initialize(queue_names, options = {})
5
- logger = options[:logger] || BBQueue::NullLogger.new
46
+ self.logger = logger = options[:logger] || BBQueue::NullLogger.new
47
+
48
+ consumer = self
6
49
 
7
50
  Stalking::Consumer.new options.merge(:logger => BBQueue::FatalLogger.new(logger)) do
8
51
  Array(queue_names).each do |queue_name|
9
52
  job queue_name do |args|
10
- BBQueue::Serializer.load(args["object"]).work
53
+ consumer.work BBQueue::Serializer.load(args["object"]), queue_name
11
54
  end
12
55
  end
13
56
 
@@ -1,3 +1,3 @@
1
1
  module BBQueue
2
- VERSION = "0.0.1"
2
+ VERSION = "0.0.2"
3
3
  end
@@ -2,18 +2,101 @@
2
2
  require File.expand_path("../../test_helper", __FILE__)
3
3
 
4
4
  class BBQueue::ConsumerTest < MiniTest::Test
5
- def test_job
6
- consumer = BBQueue::Consumer.new("default")
5
+ class ForkingConsumer < BBQueue::Consumer
6
+ attr_accessor :tempfile
7
7
 
8
- assert_equal 3, Stalking::Consumer.instances.last.run_job("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
8
+ def initialize(tempfile, *args)
9
+ self.tempfile = tempfile
10
+
11
+ super *args
12
+ end
13
+
14
+ def fork?
15
+ true
16
+ end
17
+
18
+ def before_fork
19
+ tempfile.puts "before_fork"
20
+ end
21
+
22
+ def after_fork
23
+ tempfile.puts "after_fork"
24
+ end
25
+ end
26
+
27
+ class NonForkingConsumer < ForkingConsumer
28
+ def fork?
29
+ false
30
+ end
31
+ end
32
+
33
+ class LoggingJob
34
+ attr_accessor :tempfile_path
35
+
36
+ def initialize(tempfile_path)
37
+ self.tempfile_path = tempfile_path
38
+ end
39
+
40
+ def work
41
+ open(tempfile_path, "w+") { |stream| stream.puts "work" }
42
+ end
43
+ end
44
+
45
+ class NullJob
46
+ def work
47
+ # Nothing
48
+ end
49
+ end
50
+
51
+ class RaisingJob
52
+ def work
53
+ raise "BLUBB"
54
+ end
55
+ end
56
+
57
+ def test_non_forking_job
58
+ consumer_tempfile = Tempfile.new("consumer")
59
+ consumer = NonForkingConsumer.new(consumer_tempfile, "default")
60
+
61
+ job_tempfile = Tempfile.new("job")
62
+
63
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
64
+
65
+ refute_includes File.read(consumer_tempfile.path), "before_fork"
66
+ refute_includes File.read(consumer_tempfile.path), "after_fork"
67
+
68
+ assert_includes File.read(job_tempfile.path), "work"
69
+ end
70
+
71
+ def test_forking_job
72
+ consumer_tempfile = Tempfile.new("consumer")
73
+ consumer = ForkingConsumer.new(consumer_tempfile, "default")
74
+
75
+ job_tempfile = Tempfile.new("job")
76
+
77
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
78
+
79
+ assert_includes File.read(consumer_tempfile.path), "before_fork"
80
+ assert_includes File.read(consumer_tempfile.path), "after_fork"
81
+
82
+ assert_includes File.read(job_tempfile.path), "work"
83
+ end
84
+
85
+ def test_forking_job_with_raise
86
+ logger = BBQueue::TestLogger.new
87
+ consumer = ForkingConsumer.new(Tempfile.new("consumer"), "default", :logger => logger)
88
+
89
+ assert_difference "logger.count /^error:/", 2 do
90
+ Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(RaisingJob.new) }
91
+ end
9
92
  end
10
93
 
11
94
  def test_error
12
95
  logger = BBQueue::TestLogger.new
13
96
  consumer = BBQueue::Consumer.new("default", :logger => logger)
14
97
 
15
- assert_difference "logger.count", 2 do
16
- Stalking::Consumer.instances.last.run_error("Error", "default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
98
+ assert_difference "logger.count /^error:/", 2 do
99
+ Stalking::Consumer.instances.last.run_error "Error", "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
17
100
  end
18
101
  end
19
102
 
@@ -21,8 +104,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
21
104
  logger = BBQueue::TestLogger.new
22
105
  consumer = BBQueue::Consumer.new("default", :logger => logger)
23
106
 
24
- assert_difference "logger.count" do
25
- Stalking::Consumer.instances.last.run_before("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
107
+ assert_difference "logger.count /^info:/" do
108
+ Stalking::Consumer.instances.last.run_before "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
26
109
  end
27
110
  end
28
111
 
@@ -30,8 +113,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
30
113
  logger = BBQueue::TestLogger.new
31
114
  consumer = BBQueue::Consumer.new("default", :logger => logger)
32
115
 
33
- assert_difference "logger.count" do
34
- Stalking::Consumer.instances.last.run_after("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
116
+ assert_difference "logger.count /^info:/" do
117
+ Stalking::Consumer.instances.last.run_after "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
35
118
  end
36
119
  end
37
120
  end
@@ -1,6 +1,6 @@
1
1
 
2
2
  class BBQueue::TestJob
3
- attr_accessor :x, :y
3
+ attr_accessor :x, :y, :z
4
4
 
5
5
  def initialize(x, y)
6
6
  self.x = x
@@ -12,7 +12,7 @@ class BBQueue::TestJob
12
12
  end
13
13
 
14
14
  def work
15
- x + y
15
+ self.z = x + y
16
16
  end
17
17
  end
18
18
 
@@ -1,31 +1,40 @@
1
1
 
2
+ require "logger"
3
+
2
4
  class BBQueue::TestLogger
5
+ attr_accessor :tempfile, :logger
6
+
3
7
  def initialize
4
- @messages = []
8
+ self.tempfile = Tempfile.new("logger")
9
+
10
+ logger = Logger.new(tempfile.path)
11
+ logger.formatter = proc { |severity, datetime, progname, msg| "#{severity.downcase}: #{msg}\n" }
12
+
13
+ self.logger = logger
5
14
  end
6
15
 
7
- def debug(object)
8
- @messages.push object
16
+ def debug(*args)
17
+ logger.debug(*args)
9
18
  end
10
19
 
11
- def info(object)
12
- @messages.push object
20
+ def info(*args)
21
+ logger.info(*args)
13
22
  end
14
23
 
15
- def warn(object)
16
- @messages.push object
24
+ def warn(*args)
25
+ logger.warn(*args)
17
26
  end
18
27
 
19
- def error(object)
20
- @messages.push object
28
+ def error(*args)
29
+ logger.error(*args)
21
30
  end
22
31
 
23
- def fatal(object)
24
- @messages.push object
32
+ def fatal(*args)
33
+ logger.fatal(*args)
25
34
  end
26
35
 
27
- def count
28
- @messages.size
36
+ def count(regex = /.*/)
37
+ File.read(tempfile.path).lines.grep(regex).count
29
38
  end
30
39
  end
31
40
 
@@ -2,6 +2,7 @@
2
2
  require "bbqueue"
3
3
  require "minitest"
4
4
  require "minitest/autorun"
5
+ require "tempfile"
5
6
 
6
7
  require File.expand_path("../bbqueue/stalking", __FILE__)
7
8
  require File.expand_path("../bbqueue/test_job", __FILE__)
@@ -17,9 +18,10 @@ class MiniTest::Test
17
18
  def assert_difference(expression, difference = 1, message = nil, &block)
18
19
  expressions = Array(expression)
19
20
 
20
- exps = expressions.map { |e|
21
+ exps = expressions.map do |e|
21
22
  e.respond_to?(:call) ? e : lambda { eval(e, block.binding) }
22
- }
23
+ end
24
+
23
25
  before = exps.map { |e| e.call }
24
26
 
25
27
  yield
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: bbqueue
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2014-08-06 00:00:00.000000000 Z
12
+ date: 2015-07-02 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: bundler