beaneater 0.3.3 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b2177aa90f9a3a7017a9df9295ce6105f9d40e22
-  data.tar.gz: 5f77980dcbee480728ee7723c2c67d2cd0138f25
+  metadata.gz: fe3f3a885f7fd1d205f508d48e2ce588bb56d2e4
+  data.tar.gz: 0d5578662d16fa1ad5da7f48e85f1ebf2a13d66d
 SHA512:
-  metadata.gz: 8246d9cf93f51481b0d8cdd16129aa44fe6d47c9b0ce702735a44396c364bf5f200e75e4da84857fd9b302fe89dce64f22345e0f0e79b1242c235b73bda7026c
-  data.tar.gz: ca040b82353afa029e60554d45747dc63a14bce3f8fd0e40a7d455e42873a1bebcd05441959a789c89f7dc615a8826164f300ae1e98bcf83e1fe0d68be9016ef
+  metadata.gz: 532de361e18a96f6acbbb152edc7ada7e83c2793fb494e03ea8e9f0064c6d81b20dfd296fbec26731d61a944cbdc43986492ce77f3c5e4fa30a7a3158e27a185
+  data.tar.gz: 1838fcddd8e169470a13dac5305ed0dec7b84a5fdd79289626f05a45862b4b7bce4e45cb82f455ea46ec0ed738e9003e6922b751c8f85f57aaa386e2e5b25c0f

data/CHANGELOG.md

@@ -1,5 +1,11 @@
 # CHANGELOG for Beaneater
 
+## 1.0.0 (April 26th 2015)
+
+* Beginning from version 1.0.0 the support for `Beaneater::Pool` has been dropped (@alup)
+* `Jobs#find_all` method has been removed, since it is no longer necessary after removing pool (@alup)
+* `Tubes` is now an enumerable allowing `tubes` to be handled as a collection (@Aethelflaed)
+
 ## 0.3.3 (August 16th 2014)
 
 * Fix failure when job is not defined and fix exception handling for jobs (@nicholasorenrawlings)
@@ -7,7 +13,6 @@
 * Add travis-ci badge (@tdg5)
 * Fix tests to run more reliably (@tdg5)
 
-
 ## 0.3.2 (Sept 15 2013)
 
 * Fix #29 ExpectedCrlfError name and invocation (@tdg5)

data/README.md

@@ -8,6 +8,7 @@ originally designed for reducing the latency of page views in high-volume web ap
 running time-consuming tasks asynchronously. Read the [yardocs](http://rdoc.info/github/beanstalkd/beaneater) and/or the
 [beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.md) for more details.
 
+**Important Note**: This README is **for branch 1.0.x which is under development**. Please switch to latest `0.x` branch for stable version.
 
 ## Why Beanstalk?
 
@@ -83,13 +84,25 @@ gem 'beaneater'
 
 and run `bundle install` to install the dependency.
 
+## Breaking Changes since 1.0!
+
+Starting in 1.0, we removed the concept of the `Beaneater::Pool` which introduced considerable complexity into this gem.
+
+* Beginning from version 1.0.0 the support for `Beaneater::Pool` has been dropped.
+The specific feature may be supported again in the next versions as separate module
+or through a separate gem. If you want to use the pool feature you should switch to
+0.x stable branches instead.
+* `Jobs#find_all` method has been removed, since it is no longer necessary.
+
+To manage a pool of beanstalkd instances, we'd prefer to leave the handling to the developer or other higher-level libraries.
+
 ## Quick Overview:
 
 The concise summary of how to use beaneater:
 
 ```ruby
 # Connect to pool
-@beanstalk = Beaneater::Pool.new(['localhost:11300'])
+@beanstalk = Beaneater.new('localhost:11300')
 # Enqueue jobs to tube
 @tube = @beanstalk.tubes["my-tube"]
 @tube.put '{ "key" : "foo" }', :pri => 5
@@ -118,7 +131,8 @@ Beaneater.configure do |config|
   # config.default_put_pri     = 65536
   # config.default_put_ttr     = 120
   # config.job_parser          = lambda { |body| body }
-  # config.beanstalkd_url      = ['localhost:11300']
+  # config.job_serializer      = lambda { |body| body }
+  # config.beanstalkd_url      = 'localhost:11300'
 end
 ```
 
@@ -126,18 +140,17 @@ The above options are all defaults, so only include a configuration block if you
 
 ### Connection
 
-To interact with a beanstalk queue, first establish a connection by providing a set of addresses:
+To interact with a beanstalk queue, first establish a connection by providing an address:
 
 ```ruby
-@beanstalk = Beaneater::Pool.new(['10.0.1.5:11300'])
+@beanstalk = Beaneater.new('10.0.1.5:11300')
 
-# Or if ENV['BEANSTALKD_URL] == 'localhost:11300,127.0.0.1:11300'
-@beanstalk = Beaneater::Pool.new
-@beanstalk.connections.first # => localhost:11300
-@beanstalk.connections.last # => 127.0.0.1:11300
+# Or if ENV['BEANSTALKD_URL'] == '127.0.0.1:11300'
+@beanstalk = Beaneater.new
+@beanstalk.connectiont # => localhost:11300
 ```
 
-You can conversely close and dispose of a pool at any time with:
+You can conversely close and dispose of a connection at any time with:
 
 ```ruby
 @beanstalk.close
@@ -248,6 +261,16 @@ Beanstalkd can only stores strings as job bodies, but you can easily encode your
 @tube.put({:foo => 'bar'}.to_json)
 ```
 
+Moreover, you can provide a default job serializer by setting the corresponding configuration
+option (`job_serializer`), in order to apply the encoding on each job body which
+is going to be send using the `put` command. For example, to encode a ruby object to JSON format:
+
+```ruby
+Beaneater.configure do |config|
+  config.job_serializer = lambda { |body| JSON.dump(body) }
+end
+```
+
 Each job has various metadata associated such as `priority`, `delay`, and `ttr` which can be
 specified as part of the `put` command:
 
@@ -266,7 +289,7 @@ In order to process jobs, the client should first specify the intended tubes to
 this will default to watching just the `default` tube.
 
 ```ruby
-@beanstalk = Beaneater::Pool.new(['10.0.1.5:11300'])
+@beanstalk = Beaneater.new('10.0.1.5:11300')
 @beanstalk.tubes.watch!('tube-name', 'other-tube')
 ```
 
@@ -330,13 +353,6 @@ inspected using the 'peek' commands. To find and peek at a particular job based
 # => <Beaneater::Job id=123 body="foo">
 ```
 
-You can also `find_all` jobs across all connections:
-
-```ruby
-@beanstalk.jobs.find_all(123)
-# => [<Beaneater::Job id=123 body="foo">, <Beaneater::Job id=123 body="bar">]
-```
-
 or you can peek at jobs within a tube:
 
 ```ruby
@@ -424,8 +440,9 @@ beanstalk overall:
 ```ruby
 # Get overall stats about the job processing that has occurred
 print @beanstalk.stats
-# => { 'current_connections': 1, 'current_jobs_buried': 0, ... }
-print @beanstalk.stats.current_connections
+# => #<Beaneater::StatStruct current_jobs_urgent=0, current_jobs_ready=0, current_jobs_reserved=0, current_jobs_delayed=0, current_jobs_buried=0, ...
+
+print @beanstalk.stats.current_tubes
 # => 1
 ```
 
@@ -466,3 +483,4 @@ There are other resources helpful when learning about beanstalk:
  - [Nathan Esquenazi](https://github.com/nesquena) - Contributor and co-maintainer
  - [Keith Rarick](https://github.com/kr) - Much code inspired and adapted from beanstalk-client
  - [Vidar Hokstad](https://github.com/vidarh) - Replaced telnet with correct TCP socket handling
+ - [Andreas Loupasakis](https://github.com/alup) - Improve test coverage, improve job configuration

data/examples/demo.rb

@@ -6,8 +6,7 @@ require 'beaneater'
 
 # Establish a pool of beanstalks
 puts step("Connecting to Beanstalk")
-bc = Beaneater::Pool.new('localhost')
-# bc = Beaneater::Pool.new(['localhost', 'localhost:11301', 'localhost:11302'])
+bc = Beaneater.new('localhost')
 puts bc
 
 # Print out key stats

data/lib/beaneater.rb

@@ -1,11 +1,64 @@
 require 'thread' unless defined?(Mutex)
 
-%w(version configuration errors pool_command pool connection stats tube job).each do |f|
+%w(version configuration errors connection tube job stats).each do |f|
   require "beaneater/#{f}"
 end
 
-module Beaneater
-  # Simple ruby client for beanstalkd.
+class Beaneater
+
+  # @!attribute connection
+  #   @return <Beaneater::Connection> returns the associated connection object
+  attr_reader :connection
+
+  # Initialize new instance of Beaneater
+  #
+  # @param [String] address in the form "host:port"
+  # @example
+  #   Beaneater.new('127.0.0.1:11300')
+  #
+  #   ENV['BEANSTALKD_URL'] = '127.0.0.1:11300'
+  #   @b = Beaneater.new
+  #   @b.connection.host # => '127.0.0.1'
+  #   @b.connection.port # => '11300'
+  #
+  def initialize(address)
+    @connection =  Connection.new(address)
+  end
+
+  # Returns Beaneater::Tubes object for accessing tube related functions.
+  #
+  # @return [Beaneater::Tubes] tubes object
+  # @api public
+  def tubes
+    @tubes ||= Beaneater::Tubes.new(self)
+  end
+
+  # Returns Beaneater::Jobs object for accessing job related functions.
+  #
+  # @return [Beaneater::Jobs] jobs object
+  # @api public
+  def jobs
+    @jobs ||= Beaneater::Jobs.new(self)
+  end
+
+  # Returns Beaneater::Stats object for accessing beanstalk stats.
+  #
+  # @return [Beaneater::Stats] stats object
+  # @api public
+  def stats
+    @stats ||= Stats.new(self)
+  end
+
+  # Closes the related connection
+  #
+  # @example
+  #  @beaneater_instance.close
+  #
+  def close
+    connection.close if connection
+  end
+
+  protected
 
   class << self
     # Yields a configuration block

data/lib/beaneater/configuration.rb

@@ -1,9 +1,10 @@
-module Beaneater
+class Beaneater
   class Configuration
     attr_accessor :default_put_delay   # default delay value to put a job
     attr_accessor :default_put_pri     # default priority value to put a job
     attr_accessor :default_put_ttr     # default ttr value to put a job
     attr_accessor :job_parser          # default job_parser to parse job body
+    attr_accessor :job_serializer      # default serializer for job body
     attr_accessor :beanstalkd_url      # default beanstalkd url
 
     def initialize
@@ -11,6 +12,7 @@ module Beaneater
       @default_put_pri     = 65536
       @default_put_ttr     = 120
       @job_parser          = lambda { |body| body }
+      @job_serializer      = lambda { |body| body }
       @beanstalkd_url      = nil
     end
   end # Configuration

data/lib/beaneater/connection.rb

@@ -1,10 +1,16 @@
 require 'yaml'
 require 'socket'
 
-module Beaneater
+class Beaneater
   # Represents a connection to a beanstalkd instance.
   class Connection
 
+    # Default number of retries to send a command to a connection
+    MAX_RETRIES = 3
+
+    # Default retry interval
+    DEFAULT_RETRY_INTERVAL = 1
+
     # @!attribute address
     #   @return [String] returns Beanstalkd server address
     #   @example
@@ -21,6 +27,12 @@ module Beaneater
     #   @return [Net::TCPSocket] returns connection object
     attr_reader :address, :host, :port, :connection
 
+    # @!attribute tubes_watched
+    #   @returns [Array<String>] returns currently watched tube names
+    # @!attribute tube_used
+    #   @returns [String] returns currently used tube name
+    attr_accessor :tubes_watched, :tube_used
+
     # Default port value for beanstalk connection
     DEFAULT_PORT = 11300
 
@@ -28,13 +40,23 @@ module Beaneater
     #
     # @param [String] address beanstalkd instance address.
     # @example
-    #   Beaneater::Connection.new('localhost')
-    #   Beaneater::Connection.new('localhost:11300')
+    #   Beaneater::Connection.new('127.0.0.1')
+    #   Beaneater::Connection.new('127.0.0.1:11300')
+    #
+    #   ENV['BEANSTALKD_URL'] = '127.0.0.1:11300'
+    #   @b = Beaneater.new
+    #   @b.connection.host # => '127.0.0.1'
+    #   @b.connection.port # => '11300'
     #
     def initialize(address)
-      @address = address
-      @connection = establish_connection
+      @address = address || _host_from_env || Beaneater.configuration.beanstalkd_url
       @mutex = Mutex.new
+      @tube_used = 'default'
+      @tubes_watched = ['default']
+
+      establish_connection
+    rescue
+      _raise_not_connected!
     end
 
     # Send commands to beanstalkd server via connection.
@@ -43,18 +65,19 @@ module Beaneater
     # @param [String] command Beanstalkd command
     # @return [Array<Hash{String => String, Number}>] Beanstalkd command response
     # @example
+    #   @conn = Beaneater::Connection.new
     #   @conn.transmit('bury 123')
+    #   @conn.transmit('stats')
     #
     def transmit(command, options={})
-      @mutex.synchronize do
-        if connection
+      _with_retry(options[:retry_interval], options[:init]) do
+        @mutex.synchronize do
+          _raise_not_connected! unless connection
+
           command = command.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
           connection.write(command.to_s + "\r\n")
-          res = connection.gets
-          raise_not_connected! unless res
+          res = connection.readline
           parse_response(command, res)
-        else # no connection
-          raise_not_connected!
         end
       end
     end
@@ -65,8 +88,10 @@ module Beaneater
     #  @conn.close
     #
     def close
-      @connection.close
-      @connection = nil
+      if @connection
+        @connection.close
+        @connection = nil
+      end
     end
 
     # Returns string representation of job.
@@ -79,6 +104,15 @@ module Beaneater
     end
     alias :inspect :to_s
 
+    def add_to_watched(tube_name)
+      @tubes_watched << tube_name
+      @tubes_watched.uniq
+    end
+
+    def remove_from_watched(tube_name)
+      @tubes_watched.delete(tube_name)
+    end
+
     protected
 
     # Establish a connection based on beanstalk address.
@@ -89,13 +123,11 @@ module Beaneater
     #  establish_connection('localhost:3005')
     #
     def establish_connection
-      @match = address.split(':')
-      @host, @port = @match[0], Integer(@match[1] || DEFAULT_PORT)
-      TCPSocket.new @host, @port
-    rescue Errno::ECONNREFUSED => e
-      raise NotConnected, "Could not connect to '#{@host}:#{@port}'"
-    rescue Exception => ex
-      raise NotConnected, "#{ex.class}: #{ex}"
+      @address = address.first if address.is_a?(Array)
+      match = address.split(':')
+      @host, @port = match[0], Integer(match[1] || DEFAULT_PORT)
+
+      @connection = TCPSocket.new @host, @port
     end
 
     # Parses the response and returns the useful beanstalk response.
@@ -103,11 +135,11 @@ module Beaneater
     #
     # @param [String] cmd Beanstalk command transmitted
     # @param [String] res Telnet command response
-    # @return [Array<Hash{String => String, Number}>] Beanstalk response with `status`, `id`, `body`, and `connection`
+    # @return [Array<Hash{String => String, Number}>] Beanstalk response with `status`, `id`, `body`
     # @raise [Beaneater::UnexpectedResponse] Response from beanstalk command was an error status
     # @example
     #  parse_response("delete 56", "DELETED 56\nFOO")
-    #   # => { :body => "FOO", :status => "DELETED", :id => 56, :connection => <Connection>  }
+    #   # => { :body => "FOO", :status => "DELETED", :id => 56 }
     #
     def parse_response(cmd, res)
       status = res.chomp
@@ -123,9 +155,9 @@ module Beaneater
         raise ExpectedCrlfError.new('EXPECTED_CRLF', cmd) if crlf != "\r\n"
       end
       id = body_values[1]
-      response = { :status => status, :body => body }
+      response = { :status => status }
       response[:id] = id if id
-      response[:connection] = self
+      response[:body] = body if body
       response
     end
 
@@ -136,10 +168,75 @@ module Beaneater
       Beaneater.configuration
     end
 
+    private
+
+    def _initialize_tubes
+      if @tubes_watched != ['default']
+        tubes_watched.each do |t|
+          transmit("watch #{t}", init: false)
+        end
+      end
+
+      transmit("use #{tube_used}", init: false) if @tube_used != 'default'
+    end
+
+    # Wrapper method for capturing certain failures and retry the payload block
+    #
+    # @param [Proc] block The command to execute.
+    # @param [Integer] retry_interval The time to wait before the next retry
+    # @param [Integer] tries The maximum number of tries in draining mode
+    # @return [Object] Result of the block passed
+    #
+    def _with_retry(retry_interval, init=true, tries=MAX_RETRIES, &block)
+      yield
+    rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
+      Errno::ECONNREFUSED => ex
+      _reconnect(ex, retry_interval)
+      _initialize_tubes if init
+      retry
+    rescue Beaneater::DrainingError
+      tries -= 1
+      if tries.zero?
+        close
+        raise
+      end
+      sleep(retry_interval || DEFAULT_RETRY_INTERVAL)
+      retry
+    end
+
+    # Tries to re-establish connection to the beanstalkd
+    #
+    # @param [Exception] original_exception The exception caused the retry
+    # @param [Integer] retry_interval The time to wait before the next reconnect
+    # @param [Integer] tries The maximum number of attempts to reconnect
+    def _reconnect(original_exception, retry_interval, tries=MAX_RETRIES)
+      close
+      establish_connection
+    rescue Errno::ECONNREFUSED
+      tries -= 1
+      if tries.zero?
+        _raise_not_connected!
+      end
+      sleep(retry_interval || DEFAULT_RETRY_INTERVAL)
+      retry
+    end
+
+    # The host provided by BEANSTALKD_URL environment variable, if available.
+    #
+    # @return [String] A beanstalkd host address
+    # @example
+    #  ENV['BEANSTALKD_URL'] = "localhost:1212"
+    #   # => 'localhost:1212'
+    #
+    def _host_from_env
+      ENV['BEANSTALKD_URL'].respond_to?(:length) && ENV['BEANSTALKD_URL'].length > 0 && ENV['BEANSTALKD_URL'].strip
+    end
+
     # Raises an error to be triggered when the connection has failed
     # @raise [Beaneater::NotConnected] Beanstalkd is no longer connected
-    def raise_not_connected!
-      raise NotConnected, "Connection to beanstalk '#{@host}:#{@port}' is closed!"
+    def _raise_not_connected!
+      raise Beaneater::NotConnected, "Connection to beanstalk '#{@host}:#{@port}' is closed!"
     end
+
   end # Connection
 end # Beaneater