backburner 1.1.0 → 1.2.0.pre

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9f2543596fd5e337e0b4514ca94f8b32959c3901
-  data.tar.gz: eb4133d3c373307e9399636ce8c5c4df7630e74a
+  metadata.gz: 47ef7389bf1b31ea5b36b286dbcde88b8df5ef6b
+  data.tar.gz: d218e32791da8764998ad6ca58a7bc1de0d35ae6
 SHA512:
-  metadata.gz: 58857e61ce26ff416fa6b3732839adedc2b9230886cff2fafa812c444598b264525f141a48e9eb2294ff3d8b55bd105fb00c28a89fdf1eb7250ca74022a41567
-  data.tar.gz: dcbde0b50cc908dbb6052769a546c72589585a86b9d50ec5a4ceea1c6bdc9977430cfa45321593de52bb0e2f9048dd78b151f41de524be1b730b94bc64765606
+  metadata.gz: 1f8d54cfe41edbc49995342663ee6a5a23975cca9717e72d2cc8855a2634764a635f8a766bb511044c2ea0fee196c6adb817cafd2773ef945fd78c041f23bb15
+  data.tar.gz: 0fd38fbe520aa7efc312eb468f4dda7c452c53e1f6936667e1896abf4f1b9b212775c825d2bb9b40c6a7868a586ee886720f36abc359d8c64bd5d322911457fa

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 # CHANGELOG
 
+## Version 1.2.0.pre (October 24 2015)
+
+ * FIX Replace static Beaneater connection with individual connections per worker instance/thread (@contentfree)
+ * FIX Beaneater connections try really hard to repair themselves if disconnected accidentally (@contentfree)
+ * NEW Event hook for workers: on_reconnect (@contentfree)
+
 ## Version 1.1.0 (September 14 2015)
 
  * NEW Ability to configure namespace separator (@bfolkens)

data/HOOKS.md

@@ -1,9 +1,9 @@
 # Backburner Hooks
 
-You can customize Backburner or write plugins using its hook API. 
+You can customize Backburner or write plugins using its hook API.
 In many cases you can use a hook rather than mess around with Backburner's internals.
 
-## Job Hooks 
+## Job Hooks
 
 Hooks are transparently adapted from [Resque](https://github.com/defunkt/resque/blob/master/docs/HOOKS.md), so
 if you are familiar with their hook API, now you can use nearly the same ones with beanstalkd and backburner!
@@ -26,7 +26,7 @@ There are a variety of hooks available that are triggered during the lifecycle o
 	to perform the job (but is not required to do so). It may handle exceptions
 	thrown by perform, but uncaught exceptions will be treated like regular job exceptions.
 
-* `on_retry`: Called with the retry count, the delay and the job args whenever a job is retried. 
+* `on_retry`: Called with the retry count, the delay and the job args whenever a job is retried.
 
 * `on_bury`: Called with the job args when the job is buried.
 
@@ -49,7 +49,7 @@ class SomeJob
   def self.perform(*args)
     # ...
   end
-  
+
   def self.logger
     @_logger ||= Logger.new(STDOUT)
   end
@@ -60,5 +60,16 @@ You can also setup modules to create compose-able and reusable hooks for your jo
 
 ## Worker Hooks
 
-Coming soon. What do you need here? Just let me know!
+Currently, there is just one hook:
+
+* `on_reconnect`: Called on the worker whose connection has been reset. The connection
+  is given as the argument
+
+An example:
 
+```ruby
+class MyWorker < Backburner::Worker
+  def on_reconnect(conn)
+    prepare
+  end
+end

data/README.md

@@ -113,10 +113,12 @@ The key options available are:
 | `tube_namespace`      | Prefix used for all tubes related to this backburner queue.          |
 | `namespace_separator` | Separator used for namespace and queue name                          |
 | `on_error`            | Lambda invoked with the error whenever any job in the system fails.  |
-| `default_worker`      | Worker class that will be used if no other worker is specified.      |
 | `max_job_retries`     | Integer defines how many times to retry a job before burying.        |
 | `retry_delay`         | Integer defines the base time to wait (in secs) between job retries. |
 | `retry_delay_proc`    | Lambda calculates the delay used, allowing for exponential back-off. |
+| `default_priority`    | Integer The default priority of jobs                                 |
+| `respond_timeout`     | Integer defines how long a job has to complete its task              |
+| `default_worker`      | Worker class that will be used if no other worker is specified.      |
 | `logger`              | Logger recorded to when backburner wants to report info or errors.   |
 | `primary_queue`       | Primary queue used for a job when an alternate queue is not given.   |
 | `priority_labels`     | Hash of named priority definitions for your app.                     |
@@ -158,7 +160,7 @@ class NewsletterJob
 
   # optional, defaults to respond_timeout
   def self.queue_respond_timeout
-    300 # number of seconds before job times out, 0 to avoid timeout
+    300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and beanstalkd and should be avoided
   end
 end
 ```
@@ -238,7 +240,7 @@ User.async(:queue => lambda { |user_klass| ["queue1","queue2"].sample(1).first }
 
 ### Using Async Asynchronously ###
 
-It's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in [delayed_job](https://github.com/collectiveidea/delayed_job#queuing-jobs). To accomplish this, the `Backburner::Performable` module exposes two `handle_asynchronously` convenience methods 
+It's often useful to be able to configure your app in production such that every invocation of a method is asynchronous by default as seen in [delayed_job](https://github.com/collectiveidea/delayed_job#queuing-jobs). To accomplish this, the `Backburner::Performable` module exposes two `handle_asynchronously` convenience methods
 which accept the same options as the `async` method:
 
 ```ruby
@@ -248,14 +250,14 @@ class User
   def send_welcome_email
     # ...
   end
-  
+
   # ---> For instance methods
   handle_asynchronously :send_welcome_email, queue: 'send-mail', pri: 5000, ttr: 60
 
   def self.update_recent_visitors
     # ...
   end
-  
+
   # ---> For class methods
   handle_static_asynchronously :update_recent_visitors, queue: 'long-tasks', ttr: 300
 end
@@ -471,7 +473,7 @@ end
 Note the default `max_job_retries` is 0, meaning that by default **jobs are not retried**.
 
 As jobs are retried, a progressively-increasing delay is added to give time for transient
-problems to resolve themselves. This may be configured using `retry_delay_proc`. It expects 
+problems to resolve themselves. This may be configured using `retry_delay_proc`. It expects
 an object that responds to `#call` and receives the value of `retry_delay` and the number
 of times the job has been retried already. The default is a cubic back-off, eg:
 

data/examples/demo.rb

@@ -38,7 +38,7 @@ module Tester
   end
 end
 
-# connection = Backburner::Connection.new("beanstalk://localhost")
+# connection = Backburner::Connection.new("beanstalk://127.0.0.1")
 
 Backburner.configure do |config|
   config.beanstalk_url = "beanstalk://127.0.0.1"
@@ -57,4 +57,4 @@ Tester::UserModel.async.foo("bar", "baz")
 Backburner.default_queues.concat([Tester::TestJob.queue, Tester::UserModel.queue])
 Backburner.work
 # Backburner.work("test.job")
-# Backburner.work("tester/user-model")
+# Backburner.work("tester/user-model")

data/lib/backburner/configuration.rb

@@ -19,7 +19,7 @@ module Backburner
     attr_accessor :reserve_timeout     # duration to wait to reserve on a single server
 
     def initialize
-      @beanstalk_url       = "beanstalk://localhost"
+      @beanstalk_url       = "beanstalk://127.0.0.1"
       @tube_namespace      = "backburner.worker.queue"
       @namespace_separator = "."
       @default_priority    = 65536

data/lib/backburner/connection.rb

@@ -6,32 +6,126 @@ module Backburner
 
     attr_accessor :url, :beanstalk
 
+    # If a proc is provided, it will be called (and given this connection as an
+    # argument) whenever the connection is reconnected.
+    # @example
+    #   connection.on_reconnect = lambda { |conn| puts 'reconnected!' }
+    attr_accessor :on_reconnect
+
     # Constructs a backburner connection
-    # `url` can be a string i.e 'localhost:3001' or an array of addresses.
-    def initialize(url)
+    # `url` can be a string i.e '127.0.0.1:3001' or an array of
+    # addresses (however, only the first element in the array will
+    # be used)
+    def initialize(url, &on_reconnect)
       @url = url
       @beanstalk = nil
+      @on_reconnect = on_reconnect
       connect!
     end
 
-    # Sets the delegator object to the underlying beaneater pool
-    # self.put(...)
-    def __getobj__
+    # Close the connection, if it exists
+    def close
+      @beanstalk.close if @beanstalk
+      @beanstalk = nil
       __setobj__(@beanstalk)
-      super
+    end
+
+    # Determines if the connection to Beanstalk is currently open
+    def connected?
+      begin
+        !!(@beanstalk && @beanstalk.connection && @beanstalk.connection.connection && !@beanstalk.connection.connection.closed?) # Would be nice if beaneater provided a connected? method
+      rescue => e
+        false
+      end
+    end
+
+    # Attempt to reconnect to Beanstalk. Note: the connection will not be watching
+    # or using the tubes it was before it was reconnected (as it's actually a
+    # completely new connection)
+    # @raise [Beaneater::NotConnected] If beanstalk fails to connect
+    def reconnect!
+      close
+      connect!
+      @on_reconnect.call(self) if @on_reconnect.respond_to?(:call)
+    end
+
+    # Yield to a block that will be retried several times if the connection to
+    # beanstalk goes down and is able to be re-established.
+    #
+    # @param options Hash Options. Valid options are:
+    #   :max_retries       Integer The maximum number of times the block will be yielded to.
+    #                              Defaults to 4
+    #   :on_retry          Proc    An optional proc that will be called for each retry. Will be
+    #                              called after the connection is re-established and :retry_delay
+    #                              has passed but before the block is yielded to again
+    #   :retry_delay       Float   The amount to sleep before retrying. Defaults to 1.0
+    # @raise Beaneater::NotConnected If a connection is unable to be re-established
+    def retryable(options = {}, &block)
+      options = {:max_retries => 4, :on_retry => nil, :retry_delay => 1.0}.merge!(options)
+      retry_count = options[:max_retries]
+
+      begin
+        yield
+
+      rescue Beaneater::NotConnected
+        if retry_count > 0
+          reconnect!
+          retry_count -= 1
+          sleep options[:retry_delay]
+          options[:on_retry].call if options[:on_retry].respond_to?(:call)
+          retry
+        else # stop retrying
+          raise e
+        end
+      end
     end
 
     protected
 
+    # Attempt to ensure we're connected to Beanstalk if the missing method is
+    # present in the delegate and we haven't shut down the connection on purpose
+    # @raise [Beaneater::NotConnected] If beanstalk fails to connect after multiple attempts.
+    def method_missing(m, *args, &block)
+      ensure_connected! if respond_to_missing?(m, false)
+      super
+    end
+
     # Connects to a beanstalk queue
+    # @raise Beaneater::NotConnected if the connection cannot be established
     def connect!
-      @beanstalk ||= Beaneater.new(beanstalk_addresses)
+      @beanstalk = Beaneater.new(beanstalk_addresses)
+      __setobj__(@beanstalk)
+      @beanstalk
+    end
+
+    # Attempts to ensure a connection to beanstalk is established but only if
+    # we're not connected already
+    # @param max_retries Integer The maximum number of times to attempt connecting. Defaults to 4
+    # @param retry_delay Float   The time to wait between retrying to connect. Defaults to 1.0
+    # @raise [Beaneater::NotConnected] If beanstalk fails to connect after multiple attempts.
+    # @return Connection This Connection is returned if the connection to beanstalk is open or was able to be reconnected
+    def ensure_connected!(max_retries = 4, retry_delay = 1.0)
+      return self if connected?
+
+      begin
+        reconnect!
+        return self
+
+      rescue Beaneater::NotConnected => e
+        if max_retries > 0
+          max_retries -= 1
+          sleep retry_delay
+          retry
+        else # stop retrying
+          raise e
+        end
+      end
     end
 
     # Returns the beanstalk queue addresses
     #
     # @example
-    #   beanstalk_addresses => ["localhost:11300"]
+    #   beanstalk_addresses => ["127.0.0.1:11300"]
     #
     def beanstalk_addresses
       uri = self.url.is_a?(Array) ? self.url.first : self.url
@@ -41,11 +135,11 @@ module Backburner
     # Returns a host and port based on the uri_string given
     #
     # @example
-    #   beanstalk_host_and_port("beanstalk://localhost") => "localhost:11300"
+    #   beanstalk_host_and_port("beanstalk://127.0.0.1") => "127.0.0.1:11300"
     #
     def beanstalk_host_and_port(uri_string)
       uri = URI.parse(uri_string)
-      raise(BadURL, uri_string) if uri.scheme != 'beanstalk'
+      raise(BadURL, uri_string) if uri.scheme != 'beanstalk'.freeze
       "#{uri.host}:#{uri.port || 11300}"
     end
   end # Connection

data/lib/backburner/hooks.rb

@@ -4,11 +4,11 @@ module Backburner
       # Triggers all method hooks that match the given event type with specified arguments.
       #
       # @example
-      #   invoke_hook_events(:before_enqueue, 'some', 'args')
-      #   invoke_hook_events(:after_perform, 5)
+      #   invoke_hook_events(hookable, :before_enqueue, 'some', 'args')
+      #   invoke_hook_events(hookable, :after_perform, 5)
       #
-      def invoke_hook_events(job, event, *args)
-        res = find_hook_events(job, event).map { |e| job.send(e, *args) }
+      def invoke_hook_events(hookable, event, *args)
+        res = find_hook_events(hookable, event).map { |e| hookable.send(e, *args) }
         return false if res.any? { |result| result == false }
         res
       end
@@ -20,16 +20,16 @@ module Backburner
       # original task after calling all other around blocks.
       #
       # @example
-      #   around_hook_events(:around_perform) { job.perform }
+      #   around_hook_events(hookable, :around_perform) { hookable.perform }
       #
-      def around_hook_events(job, event, *args, &block)
+      def around_hook_events(hookable, event, *args, &block)
         raise "Please pass a block to hook events!" unless block_given?
-        around_hooks = find_hook_events(job, event).reverse
+        around_hooks = find_hook_events(hookable, event).reverse
         aggregate_filter = Proc.new { |&blk| blk.call }
         around_hooks.each do |ah|
           prior_around_filter = aggregate_filter
           aggregate_filter = Proc.new do |&blk|
-            job.method(ah).call(*args) do
+            hookable.method(ah).call(*args) do
               prior_around_filter.call(&blk)
             end
           end
@@ -45,9 +45,9 @@ module Backburner
       #   find_hook_events(:before_enqueue)
       #   # => ['before_enqueue_foo', 'before_enqueue_bar']
       #
-      def find_hook_events(job, event)
-        (job.methods - Object.methods).grep(/^#{event}/).sort
+      def find_hook_events(hookable, event)
+        (hookable.methods - Object.methods).grep(/^#{event}/).sort
       end
     end
   end # Hooks
-end # Backburner
+end # Backburner

data/lib/backburner/job.rb

@@ -46,7 +46,12 @@ module Backburner
       return false unless res
       # Execute the job
       @hooks.around_hook_events(job_class, :around_perform, *args) do
-        timeout_job_after(task.ttr) { job_class.perform(*args) }
+        # We subtract one to ensure we timeout before beanstalkd does, except if:
+        #  a) ttr == 0, to support never timing out
+        #  b) ttr == 1, so that we don't accidentally set it to never time out
+        #  NB: A ttr of 1 will likely result in race conditions between
+        #  Backburner and beanstalkd and should probably be avoided
+        timeout_job_after(task.ttr > 1 ? task.ttr - 1 : task.ttr) { job_class.perform(*args) }
       end
       task.delete
       # Invoke after perform hook

data/lib/backburner/version.rb

@@ -1,3 +1,3 @@
 module Backburner
-  VERSION = "1.1.0"
+  VERSION = "1.2.0.pre"
 end

data/lib/backburner/worker.rb

@@ -27,15 +27,24 @@ module Backburner
       pri   = resolve_priority(opts[:pri] || job_class)
       delay = [0, opts[:delay].to_i].max
       ttr   = resolve_respond_timeout(opts[:ttr] || job_class)
-      res = Backburner::Hooks.invoke_hook_events(job_class, :before_enqueue, *args)
+      res   = Backburner::Hooks.invoke_hook_events(job_class, :before_enqueue, *args)
+
       return false unless res # stop if hook is false
+
       data = { :class => job_class.name, :args => args }
-      retryable_command do
-        queue = opts[:queue] && (Proc === opts[:queue] ? opts[:queue].call(job_class) : opts[:queue])
-        tube  = connection.tubes[expand_tube_name(queue || job_class)]
-        tube.put(data.to_json, :pri => pri, :delay => delay, :ttr => ttr)
+      queue = opts[:queue] && (Proc === opts[:queue] ? opts[:queue].call(job_class) : opts[:queue])
+
+      begin
+        connection = Backburner::Connection.new(Backburner.configuration.beanstalk_url)
+        connection.retryable do
+          tube = connection.tubes[expand_tube_name(queue || job_class)]
+          tube.put(data.to_json, :pri => pri, :delay => delay, :ttr => ttr)
+        end
+        Backburner::Hooks.invoke_hook_events(job_class, :after_enqueue, *args)
+      ensure
+        connection.close if connection
       end
-      Backburner::Hooks.invoke_hook_events(job_class, :after_enqueue, *args)
+
       return true
     end
 
@@ -52,21 +61,15 @@ module Backburner
       end
     end
 
-    # Returns the worker connection.
-    # @example
-    #   Backburner::Worker.connection # => <Beaneater::Connection>
-    def self.connection
-      @connection ||= Connection.new(Backburner.configuration.beanstalk_url)
-    end
-
     # List of tube names to be watched and processed
-    attr_accessor :tube_names
+    attr_accessor :tube_names, :connection
 
     # Constructs a new worker for processing jobs within specified tubes.
     #
     # @example
     #   Worker.new(['test.job'])
     def initialize(tube_names=nil)
+      @connection = new_connection
       @tube_names = self.process_tube_names(tube_names)
       register_signal_handlers!
     end
@@ -116,27 +119,34 @@ module Backburner
       compact_tube_names(tube_names)
     end
 
-    # Reserves one job within the specified queues.
-    # Pops the job off and serializes the job to JSON.
-    # Each job is performed by invoking `perform` on the job class.
+    # Performs a job by reserving a job from beanstalk and processing it
     #
     # @example
     #   @worker.work_one_job
-    #
-    def work_one_job(conn = nil)
-      conn ||= self.connection
+    # @raise [Beaneater::NotConnected] If beanstalk fails to connect multiple times.
+    def work_one_job(conn = connection)
       begin
-        job = Backburner::Job.new(conn.tubes.reserve(Backburner.configuration.reserve_timeout))
+        job = reserve_job(conn)
       rescue Beaneater::TimedOutError => e
         return
       end
+
       self.log_job_begin(job.name, job.args)
       job.process
       self.log_job_end(job.name)
+
     rescue Backburner::Job::JobFormatInvalid => e
       self.log_error self.exception_message(e)
     rescue => e # Error occurred processing job
       self.log_error self.exception_message(e)
+
+      unless job
+        self.log_error "Error occurred before we were able to assign a job. Giving up without retrying!"
+        return
+      end
+
+      # NB: There's a slight chance here that the connection to beanstalkd has
+      # gone down between the time we reserved / processed the job and here.
       num_retries = job.stats.releases
       retry_status = "failed: attempt #{num_retries+1} of #{queue_config.max_job_retries+1}"
       if num_retries < queue_config.max_job_retries # retry again
@@ -147,46 +157,23 @@ module Backburner
         job.bury
         self.log_job_end(job.name, "#{retry_status}, burying") if job_started_at
       end
-      handle_error(e, job.name, job.args, job)
-    end
 
-    # Retries the given command specified in the block several times if there is a connection error
-    # Used to execute beanstalkd commands in a retryable way
-    #
-    # @example
-    #   retryable_command { ... }
-    # @raise [Beaneater::NotConnected] If beanstalk fails to connect multiple times.
-    #
-    def self.retryable_command(max_tries=8, &block)
-      begin
-        yield
-      rescue Beaneater::NotConnected
-        retry_connection!(max_tries)
-        yield
-      end
+      handle_error(e, job.name, job.args, job)
     end
 
-    # Retries to make a connection to beanstalkd if that connection failed.
-    # @raise [Beaneater::NotConnected] If beanstalk fails to connect multiple times.
-    #
-    def self.retry_connection!(max_tries=8)
-      retry_count = 0
-      begin
-        @connection = nil
-        self.connection.stats
-      rescue Beaneater::NotConnected => e
-        if retry_count < max_tries
-          retry_count += 1
-          sleep 1
-          retry
-        else # stop retrying
-          raise e
-        end
-      end
-    end # retry_connection!
 
     protected
 
+    # Return a new connection instance
+    def new_connection
+      Connection.new(Backburner.configuration.beanstalk_url) { |conn| Backburner::Hooks.invoke_hook_events(self, :on_reconnect, conn) }
+    end
+
+    # Reserve a job from the watched queues
+    def reserve_job(conn, reserve_timeout = Backburner.configuration.reserve_timeout)
+      Backburner::Job.new(conn.tubes.reserve(reserve_timeout))
+    end
+
     # Returns a list of all tubes known within the system
     # Filtered for tubes that match the known prefix
     def all_existing_queues
@@ -195,10 +182,6 @@ module Backburner
       existing_tubes + known_queues + [queue_config.primary_queue]
     end
 
-    # Returns a reference to the beanstalk connection
-    def connection
-      self.class.connection
-    end
 
     # Handles an error according to custom definition
     # Used when processing a job that errors out