bbqueue 0.0.1 → 0.0.2

data/README.md

@@ -2,12 +2,13 @@
 
 [![Build Status](https://secure.travis-ci.org/mrkamel/bbqueue.png?branch=master)](http://travis-ci.org/mrkamel/bbqueue)
 [![Code Quality](https://codeclimate.com/github/mrkamel/bbqueue.png)](https://codeclimate.com/github/mrkamel/bbqueue)
+[![Gem Version](https://badge.fury.io/rb/bbqueue.svg)](http://badge.fury.io/rb/bbqueue)
 [![Still Maintained](http://stillmaintained.com/mrkamel/bbqueue.png)](http://stillmaintained.com/mrkamel/bbqueue)
 [![Dependency Status](https://gemnasium.com/mrkamel/bbqueue.png?travis)](https://gemnasium.com/mrkamel/bbqueue)
 
 BBQueue is an opinionated ruby gem to queue and process background jobs. Other
-gems for this purpose usually don't work with ruby objects and serialize only
-method arguments. Instead, BBQueue jobs are simple ruby objects:
+gems for this purpose usually don't work with ruby objects and serialize method
+arguments only. Instead, BBQueue jobs are simple ruby objects:
 
 ```ruby
 MyQueue.enqueue MyJob.new
@@ -16,7 +17,7 @@ MyQueue.enqueue MyJob.new
 BBQueue jobs need to fulfill the following interface:
 
 1. The object contains an instance method `#work` without any arguments
-2. The object must be serializable via `Marshal.dump` and `Marshal.load`
+2. The object (instance) must be serializable via `Marshal.dump` and `Marshal.load`
 
 ## Installation
 
@@ -36,8 +37,9 @@ Or install it yourself as:
 
 ## Usage
 
-BBQueue is currently built on top of the `stalking` gem. Therefore, you have to
-install beanstalkd.
+BBQueue is currently built on top of the
+[stalking](https://github.com/mrkamel/stalking) gem. Therefore, you have to
+install [beanstalkd](http://kr.github.io/beanstalkd/).
 
 ### Producer
 
@@ -86,8 +88,6 @@ end
 To process the enqueued jobs, create a file, e.g. jobs.rb:
 
 ```ruby
-require "bbqueue"
-
 BBQueue::Consumer.new "default"
 ```
 
@@ -96,10 +96,11 @@ and run it via:
     $ bbqueue jobs.rb
 
 BBQueue will loop through all the jobs, run them, and will then wait for new
-ones. You can pass multiple queue names and `stalking`-specific options:
+ones (no polling). You can pass multiple queue names and `stalking`-specific
+options:
 
 ```ruby
-BBQueue.new ["default", "important"], :logger => ..., :servers => ...
+BBQueue::Consumer.new ["default", "important"], :logger => ..., :servers => ...
 ```
 
 By using a separate file like `jobs.rb`, you can load your environment or do
@@ -116,6 +117,35 @@ SomeLogger.info "jobs booted into #{Rails.env.inspect} environment"
 BBQueue::Consumer.new "background", :logger => SomeLogger
 ```
 
+### Forking Consumers
+
+By default, BBQueue does not fork a process for every job, but
+it already ships with the ability to do so.
+
+To make BBQueue fork, create a custom consumer:
+
+```ruby
+class ForkingConsumer < BBQueue::Consumer
+  def fork?
+    true
+  end
+
+  def before_fork
+    # ...
+  end
+
+  def after_fork
+    # ...
+  end
+end
+```
+
+Then, start your custom consumer in favor of BBQueue's default one:
+
+```ruby
+ForkingConsumer.new ["queue1", "queue2", ...], ...
+```
+
 ## Graceful Termination
 
 Like for the `stalking` gem, you can stop a worker gracefully by sending a QUIT
@@ -123,7 +153,7 @@ signal to it. The worker will finish its current job and terminate afterwards.
 
 ## Contributing
 
-1. Fork it ( https://github.com/[my-github-username]/bbqueue/fork )
+1. Fork it ( https://github.com/mrkamel/bbqueue/fork )
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)

data/lib/bbqueue/consumer.rb

@@ -1,13 +1,56 @@
 
 module BBQueue
   class Consumer
+    attr_accessor :logger
+
+    def before_fork
+      # Nothing
+    end
+
+    def after_fork
+      # Nothing
+    end
+
+    def fork?
+      false
+    end
+
+    def fork_and_wait
+      if fork?
+        before_fork
+
+        Process.fork do
+          after_fork
+
+          yield
+        end
+
+        Process.wait
+      else
+        yield
+      end
+    end
+
+    def work(job, queue_name)
+      fork_and_wait do
+        begin
+          job.work
+        rescue Timeout::Error, StandardError => e
+          logger.error "Job #{job.inspect} on #{queue_name.inspect} failed"
+          logger.error e
+        end
+      end
+    end
+
     def initialize(queue_names, options = {})
-      logger = options[:logger] || BBQueue::NullLogger.new
+      self.logger = logger = options[:logger] || BBQueue::NullLogger.new
+
+      consumer = self
 
       Stalking::Consumer.new options.merge(:logger => BBQueue::FatalLogger.new(logger)) do
         Array(queue_names).each do |queue_name|
           job queue_name do |args|
-            BBQueue::Serializer.load(args["object"]).work
+            consumer.work BBQueue::Serializer.load(args["object"]), queue_name
           end
         end
 

data/lib/bbqueue/version.rb

@@ -1,3 +1,3 @@
 module BBQueue
-  VERSION = "0.0.1"
+  VERSION = "0.0.2"
 end

data/test/bbqueue/consumer_test.rb

@@ -2,18 +2,101 @@
 require File.expand_path("../../test_helper", __FILE__)
 
 class BBQueue::ConsumerTest < MiniTest::Test
-  def test_job
-    consumer = BBQueue::Consumer.new("default")
+  class ForkingConsumer < BBQueue::Consumer
+    attr_accessor :tempfile
 
-    assert_equal 3, Stalking::Consumer.instances.last.run_job("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
+    def initialize(tempfile, *args)
+      self.tempfile = tempfile
+
+      super *args
+    end
+
+    def fork?
+      true
+    end
+
+    def before_fork
+      tempfile.puts "before_fork"
+    end
+
+    def after_fork
+      tempfile.puts "after_fork"
+    end
+  end
+
+  class NonForkingConsumer < ForkingConsumer
+    def fork?
+      false
+    end
+  end
+
+  class LoggingJob
+    attr_accessor :tempfile_path
+
+    def initialize(tempfile_path)
+      self.tempfile_path = tempfile_path
+    end
+
+    def work
+      open(tempfile_path, "w+") { |stream| stream.puts "work" }
+    end
+  end
+
+  class NullJob
+    def work
+      # Nothing
+    end
+  end
+
+  class RaisingJob
+    def work
+      raise "BLUBB"
+    end
+  end
+
+  def test_non_forking_job
+    consumer_tempfile = Tempfile.new("consumer")
+    consumer = NonForkingConsumer.new(consumer_tempfile, "default")
+
+    job_tempfile = Tempfile.new("job")
+
+    Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
+
+    refute_includes File.read(consumer_tempfile.path), "before_fork"
+    refute_includes File.read(consumer_tempfile.path), "after_fork"
+
+    assert_includes File.read(job_tempfile.path), "work"
+  end
+
+  def test_forking_job
+    consumer_tempfile = Tempfile.new("consumer")
+    consumer = ForkingConsumer.new(consumer_tempfile, "default")
+
+    job_tempfile = Tempfile.new("job")
+
+    Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(LoggingJob.new(job_tempfile.path)) }
+
+    assert_includes File.read(consumer_tempfile.path), "before_fork"
+    assert_includes File.read(consumer_tempfile.path), "after_fork"
+
+    assert_includes File.read(job_tempfile.path), "work"
+  end
+
+  def test_forking_job_with_raise
+    logger = BBQueue::TestLogger.new
+    consumer = ForkingConsumer.new(Tempfile.new("consumer"), "default", :logger => logger)
+
+    assert_difference "logger.count /^error:/", 2 do
+      Stalking::Consumer.instances.last.run_job "default", { "object" => BBQueue::Serializer.dump(RaisingJob.new) }
+    end
   end
 
   def test_error
     logger = BBQueue::TestLogger.new
     consumer = BBQueue::Consumer.new("default", :logger => logger)
 
-    assert_difference "logger.count", 2 do
-      Stalking::Consumer.instances.last.run_error("Error", "default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
+    assert_difference "logger.count /^error:/", 2 do
+      Stalking::Consumer.instances.last.run_error "Error", "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
     end
   end
 
@@ -21,8 +104,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
     logger = BBQueue::TestLogger.new
     consumer = BBQueue::Consumer.new("default", :logger => logger)
 
-    assert_difference "logger.count" do
-      Stalking::Consumer.instances.last.run_before("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
+    assert_difference "logger.count /^info:/" do
+      Stalking::Consumer.instances.last.run_before "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
     end
   end
 
@@ -30,8 +113,8 @@ class BBQueue::ConsumerTest < MiniTest::Test
     logger = BBQueue::TestLogger.new
     consumer = BBQueue::Consumer.new("default", :logger => logger)
 
-    assert_difference "logger.count" do
-      Stalking::Consumer.instances.last.run_after("default", { "object" => BBQueue::Serializer.dump(BBQueue::TestJob.new(1, 2)) })
+    assert_difference "logger.count /^info:/" do
+      Stalking::Consumer.instances.last.run_after "default", { "object" => BBQueue::Serializer.dump(NullJob.new) }
     end
   end
 end

data/test/bbqueue/test_job.rb

@@ -1,6 +1,6 @@
 
 class BBQueue::TestJob
-  attr_accessor :x, :y
+  attr_accessor :x, :y, :z
 
   def initialize(x, y)
     self.x = x
@@ -12,7 +12,7 @@ class BBQueue::TestJob
   end
 
   def work
-    x + y
+    self.z = x + y
   end
 end
 

data/test/bbqueue/test_logger.rb

@@ -1,31 +1,40 @@
 
+require "logger"
+
 class BBQueue::TestLogger
+  attr_accessor :tempfile, :logger
+
   def initialize
-    @messages = []
+    self.tempfile = Tempfile.new("logger")
+
+    logger = Logger.new(tempfile.path)
+    logger.formatter = proc { |severity, datetime, progname, msg| "#{severity.downcase}: #{msg}\n" }
+
+    self.logger = logger
   end
 
-  def debug(object)
-    @messages.push object
+  def debug(*args)
+    logger.debug(*args)
   end
 
-  def info(object)
-    @messages.push object
+  def info(*args)
+    logger.info(*args)
   end
 
-  def warn(object)
-    @messages.push object
+  def warn(*args)
+    logger.warn(*args)
   end
 
-  def error(object)
-    @messages.push object
+  def error(*args)
+    logger.error(*args)
   end
 
-  def fatal(object)
-    @messages.push object
+  def fatal(*args)
+    logger.fatal(*args)
   end
 
-  def count
-    @messages.size
+  def count(regex = /.*/)
+    File.read(tempfile.path).lines.grep(regex).count
   end
 end
 

data/test/test_helper.rb

@@ -2,6 +2,7 @@
 require "bbqueue"
 require "minitest"
 require "minitest/autorun"
+require "tempfile"
 
 require File.expand_path("../bbqueue/stalking", __FILE__)
 require File.expand_path("../bbqueue/test_job", __FILE__)
@@ -17,9 +18,10 @@ class MiniTest::Test
   def assert_difference(expression, difference = 1, message = nil, &block)
     expressions = Array(expression)
 
-    exps = expressions.map { |e|
+    exps = expressions.map do |e|
       e.respond_to?(:call) ? e : lambda { eval(e, block.binding) }
-    }
+    end
+
     before = exps.map { |e| e.call }
 
     yield

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bbqueue
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.0.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-08-06 00:00:00.000000000 Z
+date: 2015-07-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler