beaneater 0.3.0 → 1.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 60e34aa7bfb6dc0a66ce540b609fa2ed112af52a7defcb3d7148efdbf9a8d213
+  data.tar.gz: 36c9c9cf9a1d2330243f4f8dbd664318bda4b702ecc1c160e9a288255b211764
+SHA512:
+  metadata.gz: e447f52b2790b98867e613009d7a31b3cc2af0c116656fe209514000f0a023d662a850328a3242609590f1a2fdfd9f006936f24aaf9b04d86d44f2a7b40a2dd4
+  data.tar.gz: 90eca2d3c5db4c0c96a0bfccc774d894c2d942f9ea52979f15a0a5f6b0bb39990299896a994e56f7c60537e015fc152b7fa030293c980498b545e8f5c52cb411

data/.travis.yml

@@ -0,0 +1,15 @@
+language: ruby
+rvm:
+  - 2.5
+  - 2.6
+  - 2.7
+  - 3.0
+before_install:
+  - curl -L https://github.com/kr/beanstalkd/archive/v1.9.tar.gz | tar xz -C /tmp
+  - cd /tmp/beanstalkd-1.9/
+  - make
+  - ./beanstalkd &
+  - cd $TRAVIS_BUILD_DIR
+script:
+  - bundle install
+  - bundle exec rake test:full

data/CHANGELOG.md

@@ -1,6 +1,41 @@
 # CHANGELOG for Beaneater
 
-## 0.3.1 (Unreleased)
+## 1.1.1 (April 27th 2021)
+
+* Support Ruby 3 keyword arguments (@yahonda)
+* Add CI for Ruby 3 (@yahonda)
+
+## 1.1.0 (April 25th 2021)
+
+* 'clear' behavior was failing unexpectedly, swallow issues during delete as well (@bfolkens)
+* Fix assigned but unused variables (@utilum)
+* Fix last_used tube not stored (@albb0920)
+* Fix deprecation warning in ruby 2.7 (@albb0920)
+* Fix watched tubes not restored after reconnect (@albb0920)
+* Fix keyword arguemnt warning (@albb0920)
+
+## 1.0.0 (April 26th 2015)
+
+* Beginning from version 1.0.0 the support for `Beaneater::Pool` has been dropped (@alup)
+* `Jobs#find_all` method has been removed, since it is no longer necessary after removing pool (@alup)
+* `Tubes` is now an enumerable allowing `tubes` to be handled as a collection (@Aethelflaed)
+
+## 0.3.3 (August 16th 2014)
+
+* Fix failure when job is not defined and fix exception handling for jobs (@nicholasorenrawlings)
+* Add reserve_timeout option to job processing (@nicholasorenrawlings)
+* Add travis-ci badge (@tdg5)
+* Fix tests to run more reliably (@tdg5)
+
+## 0.3.2 (Sept 15 2013)
+
+* Fix #29 ExpectedCrlfError name and invocation (@tdg5)
+
+## 0.3.1 (Jun 28 2013)
+
+* Fixes issue with "chomp" nil exception when losing connection (Thanks @simao)
+* Better handling of unknown or invalid commands during transmit
+* Raise proper CRLF exception (Thanks @carlosmoutinho)
 
 ## 0.3.0 (Jan 23 2013)
 

data/Gemfile

@@ -7,4 +7,8 @@ group :development do
   gem 'redcarpet', '~> 1'
   gem 'github-markup'
   gem 'yard'
-end
+end
+
+group :development, :test do
+  gem 'coveralls', :require => false
+end

data/README.md

@@ -1,11 +1,14 @@
 # Beaneater
+[![Build Status](https://secure.travis-ci.org/beanstalkd/beaneater.png)](http://travis-ci.org/beanstalkd/beaneater)
+[![Coverage Status](https://coveralls.io/repos/beanstalkd/beaneater/badge.png?branch=master)](https://coveralls.io/r/beanstalkd/beaneater)
 
 Beaneater is the best way to interact with beanstalkd from within Ruby.
 [Beanstalkd](http://kr.github.com/beanstalkd/) is a simple, fast work queue. Its interface is generic, but was
 originally designed for reducing the latency of page views in high-volume web applications by
 running time-consuming tasks asynchronously. Read the [yardocs](http://rdoc.info/github/beanstalkd/beaneater) and/or the
-[beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.md) for more details.
+[beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.txt) for more details.
 
+**Important Note**: This README is **for branch 1.0.x which is under development**. Please switch to latest `0.x` branch for stable version.
 
 ## Why Beanstalk?
 
@@ -81,13 +84,25 @@ gem 'beaneater'
 
 and run `bundle install` to install the dependency.
 
+## Breaking Changes since 1.0!
+
+Starting in 1.0, we removed the concept of the `Beaneater::Pool` which introduced considerable complexity into this gem.
+
+* Beginning from version 1.0.0 the support for `Beaneater::Pool` has been dropped.
+The specific feature may be supported again in the next versions as separate module
+or through a separate gem. If you want to use the pool feature you should switch to
+0.x stable branches instead.
+* `Jobs#find_all` method has been removed, since it is no longer necessary.
+
+To manage a pool of beanstalkd instances, we'd prefer to leave the handling to the developer or other higher-level libraries.
+
 ## Quick Overview:
 
 The concise summary of how to use beaneater:
 
 ```ruby
 # Connect to pool
-@beanstalk = Beaneater::Pool.new(['localhost:11300'])
+@beanstalk = Beaneater.new('localhost:11300')
 # Enqueue jobs to tube
 @tube = @beanstalk.tubes["my-tube"]
 @tube.put '{ "key" : "foo" }', :pri => 5
@@ -95,7 +110,7 @@ The concise summary of how to use beaneater:
 # Process jobs from tube
 while @tube.peek(:ready)
   job = @tube.reserve
-  puts "job value is #{job.body["key"]}!"
+  puts "job value is #{JSON.parse(job.body)["key"]}!"
   job.delete
 end
 # Disconnect the pool
@@ -116,7 +131,8 @@ Beaneater.configure do |config|
   # config.default_put_pri     = 65536
   # config.default_put_ttr     = 120
   # config.job_parser          = lambda { |body| body }
-  # config.beanstalkd_url      = ['localhost:11300']
+  # config.job_serializer      = lambda { |body| body }
+  # config.beanstalkd_url      = 'localhost:11300'
 end
 ```
 
@@ -124,18 +140,17 @@ The above options are all defaults, so only include a configuration block if you
 
 ### Connection
 
-To interact with a beanstalk queue, first establish a connection by providing a set of addresses:
+To interact with a beanstalk queue, first establish a connection by providing an address:
 
 ```ruby
-@beanstalk = Beaneater::Pool.new(['10.0.1.5:11300'])
+@beanstalk = Beaneater.new('10.0.1.5:11300')
 
-# Or if ENV['BEANSTALKD_URL] == 'localhost:11300,127.0.0.1:11300'
-@beanstalk = Beaneater::Pool.new
-@beanstalk.connections.first # => localhost:11300
-@beanstalk.connections.last # => 127.0.0.1:11300
+# Or if ENV['BEANSTALKD_URL'] == '127.0.0.1:11300'
+@beanstalk = Beaneater.new
+@beanstalk.connection # => localhost:11300
 ```
 
-You can conversely close and dispose of a pool at any time with:
+You can conversely close and dispose of a connection at any time with:
 
 ```ruby
 @beanstalk.close
@@ -246,6 +261,16 @@ Beanstalkd can only stores strings as job bodies, but you can easily encode your
 @tube.put({:foo => 'bar'}.to_json)
 ```
 
+Moreover, you can provide a default job serializer by setting the corresponding configuration
+option (`job_serializer`), in order to apply the encoding on each job body which
+is going to be send using the `put` command. For example, to encode a ruby object to JSON format:
+
+```ruby
+Beaneater.configure do |config|
+  config.job_serializer = lambda { |body| JSON.dump(body) }
+end
+```
+
 Each job has various metadata associated such as `priority`, `delay`, and `ttr` which can be
 specified as part of the `put` command:
 
@@ -264,7 +289,7 @@ In order to process jobs, the client should first specify the intended tubes to
 this will default to watching just the `default` tube.
 
 ```ruby
-@beanstalk = Beaneater::Connection.new(['10.0.1.5:11300'])
+@beanstalk = Beaneater.new('10.0.1.5:11300')
 @beanstalk.tubes.watch!('tube-name', 'other-tube')
 ```
 
@@ -328,13 +353,6 @@ inspected using the 'peek' commands. To find and peek at a particular job based
 # => <Beaneater::Job id=123 body="foo">
 ```
 
-You can also `find_all` jobs across all connections:
-
-```ruby
-@beanstalk.jobs.find_all(123)
-# => [<Beaneater::Job id=123 body="foo">, <Beaneater::Job id=123 body="bar">]
-```
-
 or you can peek at jobs within a tube:
 
 ```ruby
@@ -390,7 +408,9 @@ Processing runs the following steps:
  1. If other exception occurs, call 'bury' (error)
  1. Repeat steps 2-5
 
-The `process` command is ideally suited for a beanstalk job processing daemon.
+The `process!` command is ideally suited for a beanstalk job processing daemon. 
+Even though `process!` is intended to be a long-running process, you can stop the loop at any time
+by raising `AbortProcessingError` while processing is running.
 
 ### Handling Errors
 
@@ -409,7 +429,7 @@ are listed below:
 There are other exceptions that are less common such as `OutOfMemoryError`, `DrainingError`,
 `DeadlineSoonError`, `InternalError`, `BadFormatError`, `UnknownCommandError`,
 `ExpectedCRLFError`, `JobTooBigError`, `NotIgnoredError`. Be sure to check the
-[beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.md) for more information.
+[beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.txt) for more information.
 
 
 ### Stats
@@ -420,8 +440,9 @@ beanstalk overall:
 ```ruby
 # Get overall stats about the job processing that has occurred
 print @beanstalk.stats
-# => { 'current_connections': 1, 'current_jobs_buried': 0, ... }
-print @beanstalk.stats.current_connections
+# => #<Beaneater::StatStruct current_jobs_urgent=0, current_jobs_ready=0, current_jobs_reserved=0, current_jobs_delayed=0, current_jobs_buried=0, ...
+
+print @beanstalk.stats.current_tubes
 # => 1
 ```
 
@@ -441,7 +462,7 @@ print @beanstalk.jobs[some_job_id].stats
 # => {'age': 0, 'id': 2, 'state': 'reserved', 'tube': 'default', ... }
 ```
 
-Be sure to check the [beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.md) for
+Be sure to check the [beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.txt) for
 more details about the stats commands.
 
 ## Resources
@@ -452,11 +473,14 @@ There are other resources helpful when learning about beanstalk:
  * [Beaneater on Rubygems](https://rubygems.org/gems/beaneater)
  * [Beanstalkd homepage](http://kr.github.com/beanstalkd/)
  * [beanstalk on github](https://github.com/kr/beanstalkd)
- * [beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.md)
+ * [beanstalk protocol](https://github.com/kr/beanstalkd/blob/master/doc/protocol.txt)
  * [Backburner](https://github.com/nesquena/backburner) - Ruby job queue for Rails/Sinatra
+ * [BeanCounter](https://github.com/gemeraldbeanstalk/bean_counter) - TestUnit/MiniTest assertions and RSpec matchers for testing code that relies on Beaneater
 
 ## Contributors
 
  - [Nico Taing](https://github.com/Nico-Taing) - Creator and co-maintainer
  - [Nathan Esquenazi](https://github.com/nesquena) - Contributor and co-maintainer
- - [Keith Rarick](https://github.com/kr) - Much code inspired and adapted from beanstalk-client
+ - [Keith Rarick](https://github.com/kr) - Much code inspired and adapted from beanstalk-client
+ - [Vidar Hokstad](https://github.com/vidarh) - Replaced telnet with correct TCP socket handling
+ - [Andreas Loupasakis](https://github.com/alup) - Improve test coverage, improve job configuration

data/Rakefile

@@ -28,4 +28,6 @@ end
 YARD::Rake::YardocTask.new do |t|
   t.files   = ['lib/beaneater/**/*.rb']
   t.options = []
-end
+end
+
+task :default => 'test:full'

data/beaneater.gemspec

@@ -11,6 +11,7 @@ Gem::Specification.new do |gem|
   gem.description   = %q{Simple beanstalkd client for ruby}
   gem.summary       = %q{Simple beanstalkd client for ruby.}
   gem.homepage      = ""
+  gem.license       = 'MIT'
 
   gem.files         = `git ls-files`.split($/)
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
@@ -19,7 +20,6 @@ Gem::Specification.new do |gem|
   gem.add_development_dependency 'minitest', "~> 4.1.0"
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'mocha'
-  gem.add_development_dependency 'fakeweb'
   gem.add_development_dependency 'term-ansicolor'
   gem.add_development_dependency 'json'
 end

data/examples/demo.rb

@@ -6,8 +6,7 @@ require 'beaneater'
 
 # Establish a pool of beanstalks
 puts step("Connecting to Beanstalk")
-bc = Beaneater::Pool.new('localhost')
-# bc = Beaneater::Pool.new(['localhost', 'localhost:11301', 'localhost:11302'])
+bc = Beaneater.new('localhost')
 puts bc
 
 # Print out key stats

data/lib/beaneater/configuration.rb

@@ -1,9 +1,10 @@
-module Beaneater
+class Beaneater
   class Configuration
     attr_accessor :default_put_delay   # default delay value to put a job
     attr_accessor :default_put_pri     # default priority value to put a job
     attr_accessor :default_put_ttr     # default ttr value to put a job
     attr_accessor :job_parser          # default job_parser to parse job body
+    attr_accessor :job_serializer      # default serializer for job body
     attr_accessor :beanstalkd_url      # default beanstalkd url
 
     def initialize
@@ -11,6 +12,7 @@ module Beaneater
       @default_put_pri     = 65536
       @default_put_ttr     = 120
       @job_parser          = lambda { |body| body }
+      @job_serializer      = lambda { |body| body }
       @beanstalkd_url      = nil
     end
   end # Configuration

data/lib/beaneater/connection.rb

@@ -1,10 +1,16 @@
 require 'yaml'
 require 'socket'
 
-module Beaneater
+class Beaneater
   # Represents a connection to a beanstalkd instance.
   class Connection
 
+    # Default number of retries to send a command to a connection
+    MAX_RETRIES = 3
+
+    # Default retry interval
+    DEFAULT_RETRY_INTERVAL = 1
+
     # @!attribute address
     #   @return [String] returns Beanstalkd server address
     #   @example
@@ -21,6 +27,12 @@ module Beaneater
     #   @return [Net::TCPSocket] returns connection object
     attr_reader :address, :host, :port, :connection
 
+    # @!attribute tubes_watched
+    #   @returns [Array<String>] returns currently watched tube names
+    # @!attribute tube_used
+    #   @returns [String] returns currently used tube name
+    attr_accessor :tubes_watched, :tube_used
+
     # Default port value for beanstalk connection
     DEFAULT_PORT = 11300
 
@@ -28,13 +40,23 @@ module Beaneater
     #
     # @param [String] address beanstalkd instance address.
     # @example
-    #   Beaneater::Connection.new('localhost')
-    #   Beaneater::Connection.new('localhost:11300')
+    #   Beaneater::Connection.new('127.0.0.1')
+    #   Beaneater::Connection.new('127.0.0.1:11300')
+    #
+    #   ENV['BEANSTALKD_URL'] = '127.0.0.1:11300'
+    #   @b = Beaneater.new
+    #   @b.connection.host # => '127.0.0.1'
+    #   @b.connection.port # => '11300'
     #
     def initialize(address)
-      @address = address
-      @connection = establish_connection
+      @address = address || _host_from_env || Beaneater.configuration.beanstalkd_url
       @mutex = Mutex.new
+      @tube_used = 'default'
+      @tubes_watched = ['default']
+
+      establish_connection
+    rescue
+      _raise_not_connected!
     end
 
     # Send commands to beanstalkd server via connection.
@@ -43,19 +65,21 @@ module Beaneater
     # @param [String] command Beanstalkd command
     # @return [Array<Hash{String => String, Number}>] Beanstalkd command response
     # @example
+    #   @conn = Beaneater::Connection.new
     #   @conn.transmit('bury 123')
+    #   @conn.transmit('stats')
     #
-    def transmit(command, options={})
-      @mutex.lock
-      if connection
-        command = command.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
-        connection.write(command+"\r\n")
-        parse_response(command, connection.gets)
-      else # no connection
-        raise NotConnected, "Connection to beanstalk '#{@host}:#{@port}' is closed!" unless connection
+    def transmit(command, **options)
+      _with_retry(**options.slice(:retry_interval, :init)) do
+        @mutex.synchronize do
+          _raise_not_connected! unless connection
+
+          command = command.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
+          connection.write(command.to_s + "\r\n")
+          res = connection.readline
+          parse_response(command, res)
+        end
       end
-    ensure
-      @mutex.unlock
     end
 
     # Close connection with beanstalkd server.
@@ -64,8 +88,10 @@ module Beaneater
     #  @conn.close
     #
     def close
-      @connection.close
-      @connection = nil
+      if @connection
+        @connection.close
+        @connection = nil
+      end
     end
 
     # Returns string representation of job.
@@ -78,23 +104,30 @@ module Beaneater
     end
     alias :inspect :to_s
 
+    def add_to_watched(tube_name)
+      @tubes_watched << tube_name
+      @tubes_watched.uniq
+    end
+
+    def remove_from_watched(tube_name)
+      @tubes_watched.delete(tube_name)
+    end
+
     protected
 
     # Establish a connection based on beanstalk address.
     #
     # @return [Net::TCPSocket] connection for specified address.
-    # @raise [Beanstalk::NotConnected] Could not connect to specified beanstalkd instance.
+    # @raise [Beaneater::NotConnected] Could not connect to specified beanstalkd instance.
     # @example
     #  establish_connection('localhost:3005')
     #
     def establish_connection
-      @match = address.split(':')
-      @host, @port = @match[0], Integer(@match[1] || DEFAULT_PORT)
-      TCPSocket.new @host, @port
-    rescue Errno::ECONNREFUSED => e
-      raise NotConnected, "Could not connect to '#{@host}:#{@port}'"
-    rescue Exception => ex
-      raise NotConnected, "#{ex.class}: #{ex}"
+      @address = address.first if address.is_a?(Array)
+      match = address.split(':')
+      @host, @port = match[0], Integer(match[1] || DEFAULT_PORT)
+
+      @connection = TCPSocket.new @host, @port
     end
 
     # Parses the response and returns the useful beanstalk response.
@@ -102,11 +135,11 @@ module Beaneater
     #
     # @param [String] cmd Beanstalk command transmitted
     # @param [String] res Telnet command response
-    # @return [Array<Hash{String => String, Number}>] Beanstalk response with `status`, `id`, `body`, and `connection`
+    # @return [Array<Hash{String => String, Number}>] Beanstalk response with `status`, `id`, `body`
     # @raise [Beaneater::UnexpectedResponse] Response from beanstalk command was an error status
     # @example
     #  parse_response("delete 56", "DELETED 56\nFOO")
-    #   # => { :body => "FOO", :status => "DELETED", :id => 56, :connection => <Connection>  }
+    #   # => { :body => "FOO", :status => "DELETED", :id => 56 }
     #
     def parse_response(cmd, res)
       status = res.chomp
@@ -119,12 +152,12 @@ module Beaneater
         raw_body = connection.read(bytes_size)
         body = status == 'OK' ? YAML.load(raw_body) : config.job_parser.call(raw_body)
         crlf = connection.read(2) # \r\n
-        raise ExpectedCRLFError if crlf != "\r\n"
+        raise ExpectedCrlfError.new('EXPECTED_CRLF', cmd) if crlf != "\r\n"
       end
       id = body_values[1]
-      response = { :status => status, :body => body }
+      response = { :status => status }
       response[:id] = id if id
-      response[:connection] = self
+      response[:body] = body if body
       response
     end
 
@@ -135,5 +168,77 @@ module Beaneater
       Beaneater.configuration
     end
 
+    private
+
+    def _initialize_tubes
+      if @tubes_watched != ['default']
+        tubes_watched.each do |t|
+          transmit("watch #{t}", init: false)
+        end
+
+        transmit("ignore default", init: false)
+      end
+
+      transmit("use #{tube_used}", init: false) if @tube_used != 'default'
+    end
+
+    # Wrapper method for capturing certain failures and retry the payload block
+    #
+    # @param [Proc] block The command to execute.
+    # @param [Integer] retry_interval The time to wait before the next retry
+    # @param [Integer] tries The maximum number of tries in draining mode
+    # @return [Object] Result of the block passed
+    #
+    def _with_retry(retry_interval: DEFAULT_RETRY_INTERVAL, init: true, tries: MAX_RETRIES, &block)
+      yield
+    rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
+      Errno::ECONNREFUSED => ex
+      _reconnect(ex, retry_interval)
+      _initialize_tubes if init
+      retry
+    rescue Beaneater::DrainingError
+      tries -= 1
+      if tries.zero?
+        close
+        raise
+      end
+      sleep(retry_interval)
+      retry
+    end
+
+    # Tries to re-establish connection to the beanstalkd
+    #
+    # @param [Exception] original_exception The exception caused the retry
+    # @param [Integer] retry_interval The time to wait before the next reconnect
+    # @param [Integer] tries The maximum number of attempts to reconnect
+    def _reconnect(original_exception, retry_interval, tries=MAX_RETRIES)
+      close
+      establish_connection
+    rescue Errno::ECONNREFUSED
+      tries -= 1
+      if tries.zero?
+        _raise_not_connected!
+      end
+      sleep(retry_interval || DEFAULT_RETRY_INTERVAL)
+      retry
+    end
+
+    # The host provided by BEANSTALKD_URL environment variable, if available.
+    #
+    # @return [String] A beanstalkd host address
+    # @example
+    #  ENV['BEANSTALKD_URL'] = "localhost:1212"
+    #   # => 'localhost:1212'
+    #
+    def _host_from_env
+      ENV['BEANSTALKD_URL'].respond_to?(:length) && ENV['BEANSTALKD_URL'].length > 0 && ENV['BEANSTALKD_URL'].strip
+    end
+
+    # Raises an error to be triggered when the connection has failed
+    # @raise [Beaneater::NotConnected] Beanstalkd is no longer connected
+    def _raise_not_connected!
+      raise Beaneater::NotConnected, "Connection to beanstalk '#{@host}:#{@port}' is closed!"
+    end
+
   end # Connection
 end # Beaneater

data/lib/beaneater/errors.rb

@@ -1,4 +1,4 @@
-module Beaneater
+class Beaneater
   # Raises when a beanstalkd instance is no longer accessible.
   class NotConnected < RuntimeError; end
   # Raises when the tube name specified is invalid.
@@ -66,11 +66,11 @@ module Beaneater
   # Raises when a command was sent that is unknown.
   class UnknownCommandError < UnexpectedResponse; end
   # Raises when command does not have proper CRLF suffix.
-  class ExpectedCRLFError < UnexpectedResponse; end
+  class ExpectedCrlfError < UnexpectedResponse; end
   # Raises when the body of a job was too large.
   class JobTooBigError < UnexpectedResponse; end
-  # Raises when a job was attempted to be reserved but the timeout occured.
+  # Raises when a job was attempted to be reserved but the timeout occurred.
   class TimedOutError < UnexpectedResponse; end
   # Raises when a tube could not be ignored because it is the last watched tube.
   class NotIgnoredError < UnexpectedResponse; end
-end
+end