beaneater 1.1.4 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 27dbb64059340085f2e89899e550b2329db2043e96491ae2561b2dd4f2bb34ea
-  data.tar.gz: 3bf0277effe249cbbcd95e6643c6d5f7c8bd7a3efeee3a3e86f2cd6c53b59402
+  metadata.gz: 7397bf5f83a1ba3ff93d90fb1ee7718d8dcaf90ac45ecdedd236fe4fe666c5eb
+  data.tar.gz: e3f0776719d39a2eb235a921b505bddb3af25be0c915718be3b24a46dd0afd0b
 SHA512:
-  metadata.gz: 65c0e86ba162862f43ec6bf3d4c0f45b1e30821f7bd7a891d8833be1312d256d2ec6999406bee08c5bf0e151ce2a599a7274c9179f3b64563d6168fc02f39888
-  data.tar.gz: ab0f650fade61572fb3b1a8c57c3e0c15634e826f1a5cd2635ef07e4270ce472f8b53c02488028b14c25a294498b3b494854d3f52e798109a61ef7cb957bc674
+  metadata.gz: 9e6b026dbb985f693b2fb0c494084223f86100850210cef83af8d6c7a951e6f81b2dd620262d5f28fa4a18ec8c26542409c45fd9404199dba766470af8265af1
+  data.tar.gz: a73f631654a1147afcaed139b24b8ec1748fde51b02ff9e84492e8ccdcae81cc84177951b5faebf5d8c06189188b7448e21c346f2b2650219467bcbe1f62b1c8

data/.github/workflows/ruby.yml

@@ -22,10 +22,10 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        ruby-version: ['2.6', '2.7', '3.0', '3.1']
+        ruby-version: ['2.7', '3.0', '3.1', '3.2', '3.3', '3.4', '4.0']
 
     steps:
-    - uses: actions/checkout@v3
+    - uses: actions/checkout@v4
     - name: Download & Extract beanstalkd
       run: curl -L https://github.com/beanstalkd/beanstalkd/archive/refs/tags/v1.13.tar.gz | tar xz
     - name: Make beanstalkd
@@ -40,3 +40,5 @@ jobs:
       run: ./beanstalkd-1.13/beanstalkd &
     - name: Run tests
       run: bundle exec rake test:full
+      env:
+        BEANSTALKD_BIN: ${{ github.workspace }}/beanstalkd-1.13/beanstalkd

data/CHANGELOG.md

@@ -1,5 +1,21 @@
 # CHANGELOG for Beaneater
 
+## 1.2.0 (October 31 2026)
+
+* `Jobs#process!` buries a job whose body the `job_parser` rejects (`JSON.parse` on a non-JSON job, say); it used to stay reserved until its TTR ran out and then fail again, for ever. A parser that raises `AbortProcessingError` has its job buried and stops the loop, as a handler's abort does. The job is not buried when the parser raises `NotConnected` while this client's connection is gone (`process!` raises it, and beanstalkd releases the job), an exception that isn't a `StandardError` (`process!` raises it, and the job stays reserved until its TTR runs out or the connection closes), or one that can't be copied (raised untagged, so `process!` can't tell which job it came from: the job stays reserved and `process!` keeps failing on it; make parser exceptions copyable). A plain `tubes.reserve` raises a copy of the parser's error with the same class, message, cause and backtrace (a frozen exception, for which Ruby records no backtrace, gets one that starts in beaneater), so `rescue JSON::ParserError` keeps working; the copy is also a `Beaneater::JobParseFailure`, whose `#job_id` names the job, still reserved by the connection (@nesquena)
+* `close` while a command is waiting on the connection (a `reserve`, say), whether from a signal handler or from another thread, now makes that command raise `NotConnected`; 1.1.x raised `IOError`, and on Ruby 3.3 and 3.4 a `close` from a signal handler could crash the process. A `close` from a `job_parser` lets its command finish, then closes the socket (@nesquena)
+* Drop the connection when a command is interrupted mid-reply (for example by `Timeout.timeout`), so the next command no longer reads the stale reply (reported by @tdegrunt)
+* Honour `stop!` while `process!` is waiting for a job with `reserve_timeout` set (reported by @martint17r)
+* Keep `process!` running when a handler deletes or releases its job, or overruns its TTR, and then raises; it used to stop with `NotFoundError` or `JobNotReserved`. A handler that raises one of those (or `TimedOutError`) while its job is still reserved now has the job buried like any other failure, instead of leaving it reserved and forgotten, which made `process!` reserve in a tight loop of `DEADLINE_SOON` replies near the job's TTR. To have such a job released for another try instead, add the error to the handler's `retry_on` (`jobs.register(tube, :retry_on => [Beaneater::NotFoundError])`); it is released up to `max_retries` times, then buried (reported by @pagojo)
+* Reconnect on the next command after the connection was lost, restoring the used and watched tubes (while beanstalkd is down each command now retries for about 2 seconds before raising `NotConnected`, and gives up after 3 reconnects when every new connection is dropped at once; set `connect_timeout` in web apps that enqueue during requests), and make `process!` raise `NotConnected` instead of looping while its beanstalkd is down (a handler that raises `NotConnected` for another server still has its job buried, as before) (reported by @b-sridhar and @Meekohi)
+* Test on Ruby 2.7 to 3.4 and 4.0 in CI (dropping 2.6), and update `actions/checkout` to v4 (@nesquena)
+* Require Ruby 2.7 or newer, declared in the gemspec, so older Rubies get a clear install-time error instead of failing at runtime (the code needs at least 2.5 for `Hash#slice`), #83 (@nesquena; reported by @hkmaly)
+* Make `read_timeout` and `write_timeout` take effect: they set `SO_RCVTIMEO`/`SO_SNDTIMEO`, which have no effect on Ruby 3 (Ruby waits on the socket itself), so a server that stopped answering still blocked forever. Beaneater now enforces both itself, adds `read_timeout` to `reserve(timeout)`'s own wait, never cuts off a plain `reserve`, and on timeout raises the new `Beaneater::SocketTimeout` (a `NotConnected`) and drops the connection, so the next command reconnects. `read_timeout` and `write_timeout` may be numeric strings. A `SocketTimeout` on `put` is ambiguous (beanstalkd may have stored the job), so retrying it can enqueue the job twice (@nesquena)
+* Fix a raising `job_parser` (e.g. `JSON.parse` on a non-JSON job) leaving the connection one reply out of step, so later commands returned the wrong replies (@nesquena)
+* Fix stats and tube lists mangling values since 1.1.2: `-`, `:` and `~` were rewritten to `_`, so `job.tube` for tube `email-jobs` was `"email_jobs"` and `Jobs#process!` buried its jobs; tube names that look like dates or numbers (`2026-10-30`, `1.10`) are now returned as strings instead of raising or being converted (@nesquena)
+* Add `connect_timeout` (Ruby 3+), `resolv_timeout` (Ruby 3+), `read_timeout`, and `write_timeout` configuration options (@rveznaver)
+* Fix `StatStruct#[]` raising a `SyntaxError` when looking up hyphenated beanstalkd keys (e.g. `stats['current-jobs-ready']`); keys are now underscored before lookup to match how they are stored (@dkam)
+
 ## 1.1.4 (March 6 2026)
 
 * Add frozen string literal comments for Ruby 3.4 compatibility (@nearapogee)

data/README.md

@@ -133,11 +133,21 @@ Beaneater.configure do |config|
   # config.job_parser          = lambda { |body| body }
   # config.job_serializer      = lambda { |body| body }
   # config.beanstalkd_url      = 'localhost:11300'
+  # config.connect_timeout     = nil
+  # config.resolv_timeout      = nil
+  # config.read_timeout        = nil
+  # config.write_timeout       = nil
 end
 ```
 
 The above options are all defaults, so only include a configuration block if you need to make changes.
 
+`connect_timeout` and `resolv_timeout` are passed through to `TCPSocket.new` on Ruby 3.0 and newer (ignored on Ruby < 3.0 for compatibility). `read_timeout` bounds how long beaneater waits for beanstalkd's reply to a command; for `reserve(timeout)` it is added to the reserve's own timeout, and a plain `reserve` (no timeout) still waits as long as it takes for a job. `write_timeout` bounds how long sending a command may take. When either runs out, `Beaneater::SocketTimeout` (a `Beaneater::NotConnected`) is raised and the connection is dropped; the next command reconnects. Timeouts are in seconds; `read_timeout` and `write_timeout` may also be numeric strings (from `ENV`, say).
+
+A `SocketTimeout` on `put` does not tell you whether beanstalkd stored the job: the command may have reached the server and only the reply been lost. Retrying the `put` can therefore enqueue the job twice, so retry a timed-out `put` only if running the job twice is harmless.
+
+While beanstalkd is unreachable, every command tries to reconnect three times, one second apart, before raising `Beaneater::NotConnected`, so each command takes about 2 seconds to fail (longer per attempt when connecting itself hangs, as with an unreachable host). Threads sharing a client pause between their retries side by side, but the connection attempts themselves still run one at a time. Apps that enqueue jobs inside web requests should set `connect_timeout` and expect this delay while beanstalkd is down.
+
 ### Connection
 
 To interact with a beanstalk queue, first establish a connection by providing an address:
@@ -412,6 +422,23 @@ The `process!` command is ideally suited for a beanstalk job processing daemon.
 Even though `process!` is intended to be a long-running process, you can stop the loop at any time
 by raising `AbortProcessingError` while processing is running.
 
+If the `job_parser` raises for a job (a body that isn't JSON, say), `process!` buries that job and
+goes on; if it raises `AbortProcessingError`, `process!` buries the job and stops, as for a
+handler's abort. The job is not buried when the parser raises `NotConnected` while this client's
+connection is gone (`process!` raises it, and beanstalkd releases the job), an exception that isn't
+a `StandardError` (`process!` raises it, and the job stays reserved until its TTR runs out or the
+connection closes), or one that can't be copied (raised untagged, so `process!` can't tell which job
+it came from: the job stays reserved and `process!` keeps failing on it; make parser exceptions
+copyable). A direct `tubes.reserve` raises a copy of
+the parser's error (same class, message, cause and backtrace; a frozen exception, for which Ruby
+records no backtrace, gets one that starts in beaneater) that is also a
+`Beaneater::JobParseFailure`: its `job_id` names the job, still reserved by your connection, so
+you can bury, release or delete it.
+
+If beanstalkd goes away and reconnecting fails, `process!` raises `Beaneater::NotConnected`.
+The client reconnects on its next command, so calling `process!` again (for example from a
+supervisor loop) resumes once the server is back.
+
 ### Handling Errors
 
 While using Beaneater, certain errors may be encountered. Errors are encountered when
@@ -424,6 +451,7 @@ are listed below:
 | Beaneater::InvalidTubeName  | Specified tube name for use or watch is not valid.    |
 | Beaneater::NotFoundError    | Specified job or tube could not be found.             |
 | Beaneater::TimedOutError    | Job could not be reserved within time specified.      |
+| Beaneater::SocketTimeout    | No reply within `read_timeout` (or the write exceeded `write_timeout`); the connection is dropped and the next command reconnects. A subclass of `NotConnected`, not to be confused with `TimedOutError`. |
 | Beaneater::JobNotReserved   | Job has not been reserved and action cannot be taken. |
 
 There are other exceptions that are less common such as `OutOfMemoryError`, `DrainingError`,

data/beaneater.gemspec

@@ -18,6 +18,7 @@ Gem::Specification.new do |gem|
   gem.executables   = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})
   gem.require_paths = ["lib"]
+  gem.required_ruby_version = '>= 2.7'
   gem.add_development_dependency 'minitest', "~> 4.1.0"
   gem.add_development_dependency 'rake'
   gem.add_development_dependency 'mocha'

data/lib/beaneater/configuration.rb

@@ -8,6 +8,10 @@ class Beaneater
     attr_accessor :job_parser          # default job_parser to parse job body
     attr_accessor :job_serializer      # default serializer for job body
     attr_accessor :beanstalkd_url      # default beanstalkd url
+    attr_accessor :connect_timeout     # TCP connect timeout in seconds
+    attr_accessor :resolv_timeout      # DNS resolve timeout in seconds
+    attr_accessor :read_timeout        # seconds to wait for a reply (added to reserve-with-timeout's own timeout; plain reserve is never cut off)
+    attr_accessor :write_timeout       # seconds to wait for a command to be written
 
     def initialize
       @default_put_delay   = 0
@@ -16,6 +20,10 @@ class Beaneater
       @job_parser          = lambda { |body| body }
       @job_serializer      = lambda { |body| body }
       @beanstalkd_url      = nil
+      @connect_timeout     = nil
+      @resolv_timeout      = nil
+      @read_timeout        = nil
+      @write_timeout       = nil
     end
   end # Configuration
 end # Beaneater

data/lib/beaneater/connection.rb

@@ -2,6 +2,8 @@
 
 require 'yaml'
 require 'socket'
+require 'io/wait'
+require 'monitor'
 
 class Beaneater
   # Represents a connection to a beanstalkd instance.
@@ -13,6 +15,15 @@ class Beaneater
     # Default retry interval
     DEFAULT_RETRY_INTERVAL = 1
 
+    # Fields of the stats responses that are text even when they look numeric
+    # (a tube named "1.12", a server id made only of digits). beanstalkd 1.13
+    # quotes most of these, 1.12 does not.
+    STRING_STAT_FIELDS = {
+      'stats'      => %w(version id hostname os platform),
+      'stats-job'  => %w(tube state),
+      'stats-tube' => %w(name)
+    }.freeze
+
     # @!attribute address
     #   @return [String] returns Beanstalkd server address
     #   @example
@@ -52,7 +63,10 @@ class Beaneater
     #
     def initialize(address)
       @address = address || _host_from_env || Beaneater.configuration.beanstalkd_url
-      @mutex = Mutex.new
+      # Held for each attempt at a command (reconnecting and restoring the tubes included), not
+      # across the pauses between attempts. Reentrant: a job_parser that uses this client runs
+      # while its command holds the lock.
+      @mutex = Monitor.new
       @tube_used = 'default'
       @tubes_watched = ['default']
 
@@ -72,16 +86,27 @@ class Beaneater
     #   @conn.transmit('stats')
     #
     def transmit(command, **options)
-      _with_retry(**options.slice(:retry_interval, :init)) do
-        @mutex.synchronize do
-          _raise_not_connected! unless connection
-
-          command = command.dup.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
-          connection.write(command.to_s + "\r\n")
-          res = connection.readline
-          parse_response(command, res)
-        end
+      state = {
+        :retry_interval => options[:retry_interval] || DEFAULT_RETRY_INTERVAL, # nil too
+        :init           => options.fetch(:init, true),
+        :reconnects     => 0,
+        :connect_tries  => MAX_RETRIES,
+        :draining_tries => MAX_RETRIES,
+        :connecting     => false
+      }
+      reentrant = @mutex.mon_owned?
+      loop do
+        done, value = @mutex.synchronize { _attempt(command, state) }
+        return value if done
+        # Pause before the next attempt with the lock released, so other threads go on and
+        # Timeout or Thread#raise can stop this one. A reentrant call (from a job_parser) still
+        # holds its outer command's lock here.
+        sleep(value)
       end
+    ensure
+      # #close during a command (from its job_parser, a signal handler or another thread) only
+      # shut the socket down; close it once the command is over and nothing waits on it
+      @mutex.synchronize { _drop_connection if @closed && @connection } if @closed && !reentrant
     end
 
     # Close connection with beanstalkd server.
@@ -90,10 +115,31 @@ class Beaneater
     #  @conn.close
     #
     def close
-      if @connection
-        @connection.close
-        @connection = nil
+      @closed = true
+      socket = @connection
+      if @mutex.mon_locked? && @connection_pid == Process.pid
+        # A thread (this one or another) is inside a command and may be waiting on the socket.
+        # IO#close from a signal handler while any thread waits on that IO can crash Ruby 3.3
+        # and 3.4 (segfault in IO#wait_readable) or raise ThreadError. Shut it down instead:
+        # the wait ends with end-of-file, the command raises NotConnected, and its transmit
+        # closes the socket on its way out. Only in the process that opened the connection: a
+        # child forked during a command inherits the held lock, and shutting down would cut
+        # the parent's connection; the child just closes its own copy below.
+        socket.shutdown(Socket::SHUT_RDWR) if socket && !socket.closed?
+      else
+        # ceiling: a thread can take the lock between the check above and this close and start
+        # waiting on the socket. A close from a signal handler then raises ThreadError on Ruby
+        # 3.3 and 3.4 (ignored below); the waiting thread gets a TypeError from Ruby, and Ruby
+        # keeps the fd open (and the server that connection's pending command) until garbage
+        # collection. On 3.1, 3.2 and 4.0, and from a plain thread, the waiter gets IOError.
+        # Closing under the lock would shut the window, but a signal handler cannot wait for
+        # the lock.
+        _drop_connection
       end
+    rescue ThreadError
+      nil # IO#close refused in a signal handler (see the ceiling above)
+    rescue SystemCallError, IOError
+      nil # already disconnected, or closed by another thread meanwhile
     end
 
     # Returns string representation of job.
@@ -129,7 +175,17 @@ class Beaneater
       match = address.split(':')
       @host, @port = match[0], Integer(match[1] || DEFAULT_PORT)
 
-      @connection = TCPSocket.new @host, @port
+      tcp_opts = { connect_timeout: config.connect_timeout, resolv_timeout: config.resolv_timeout }.compact
+
+      socket = if RUBY_VERSION >= "3.0" && tcp_opts.any?
+        TCPSocket.new(@host, @port, **tcp_opts)
+      else
+        TCPSocket.new(@host, @port)
+      end
+
+      @read_buffer = String.new # binary; bytes read but not yet consumed
+      @connection = socket
+      @connection_pid = Process.pid # a forked child shares the socket; see #close
     end
 
     # Parses the response and returns the useful beanstalk response.
@@ -151,15 +207,17 @@ class Beaneater
       body = nil
       if ['OK','FOUND', 'RESERVED'].include?(status)
         bytes_size = body_values[-1].to_i
-        raw_body = connection.read(bytes_size)
+        raw_body = _read_bytes(bytes_size, cmd)
+        # Consume the trailing CRLF before parsing, so a parser that raises
+        # leaves the stream at the start of the next reply.
+        crlf = _read_bytes(2, cmd) # \r\n
+        raise ExpectedCrlfError.new('EXPECTED_CRLF', cmd) if crlf != "\r\n"
+        @reply_read = true
         body = if status == 'OK'
-          psych_v4_valid_body = raw_body.gsub(/^(.*?): (.*)$/) { "#{$1}: #{$2.gsub(/[\:\-\~]/, '_')}" }
-          YAML.load(psych_v4_valid_body)
+          parse_yaml_body(cmd, raw_body)
         else
-          config.job_parser.call(raw_body)
+          _parse_job(status, body_values[1], raw_body)
         end
-        crlf = connection.read(2) # \r\n
-        raise ExpectedCrlfError.new('EXPECTED_CRLF', cmd) if crlf != "\r\n"
       end
       id = body_values[1]
       response = { :status => status }
@@ -168,6 +226,57 @@ class Beaneater
       response
     end
 
+    # Parses the YAML body of an OK response (stats, stats-job, stats-tube,
+    # list-tubes, list-tubes-watched). beanstalkd emits only a "---" line
+    # followed by "key: value" lines or "- item" lines, so the body is parsed
+    # directly rather than through a YAML loader, which would turn tube names
+    # such as "2022-09-30", "1.12" or "true" into dates, numbers or booleans.
+    #
+    # @param [String] cmd Beanstalk command transmitted
+    # @param [String] raw_body Response body
+    # @return [Hash{String => String, Integer, Float, Boolean}, Array<String>] Parsed body
+    # @example
+    #  parse_yaml_body("stats-job 1", "---\nid: 1\ntube: email-jobs\n")
+    #   # => { "id" => 1, "tube" => "email-jobs" }
+    #
+    def parse_yaml_body(cmd, raw_body)
+      lines = raw_body.dup.force_encoding(Encoding::UTF_8).each_line.map(&:chomp)
+      lines.shift if lines.first == '---'
+      if lines.first && lines.first.start_with?('- ')
+        lines.map { |line| _stat_value(line[2..-1], true) }
+      else
+        text_fields = STRING_STAT_FIELDS[cmd.to_s.split(' ').first] || []
+        lines.each_with_object({}) do |line, hash|
+          key, value = line.split(': ', 2)
+          hash[key] = _stat_value(value.to_s, text_fields.include?(key)) unless key.to_s.empty?
+        end
+      end
+    end
+
+    # Runs the job_parser on a job's body. If it raises for a job this connection has just
+    # reserved, a copy of the parser's exception is raised instead (same class, message,
+    # backtrace and cause, so `rescue JSON::ParserError` still works), tagged with
+    # Beaneater::JobParseFailure, which gives the job's id: Jobs#process! uses it to bury the job
+    # instead of leaving it reserved until its TTR. A copy, so that an exception the parser
+    # froze or raises again later is never changed or tagged with an old id.
+    def _parse_job(status, id, raw_body)
+      config.job_parser.call(raw_body)
+    rescue StandardError => ex
+      raise unless status == 'RESERVED'
+      begin
+        tagged = ex.clone(:freeze => false)
+        tagged.extend(JobParseFailure)
+        tagged.job_id = id
+        cause = ex.cause
+      rescue StandardError
+        tagged = nil # it can't be copied (an initialize_clone or initialize_copy of its own)
+      end
+      # Re-raised out here, where $! is still ex: raised inside the rescue above, a frozen ex
+      # would get the copying error as its cause and fail with "circular causes".
+      raise ex unless tagged
+      raise tagged, :cause => (cause if cause.is_a?(Exception))
+    end
+
     # Returns configuration options for beaneater
     #
     # @return [Beaneater::Configuration] configuration object
@@ -177,57 +286,111 @@ class Beaneater
 
     private
 
+    # Sends one command and reads its whole reply, without retrying.
+    def _exchange(command)
+      completed = false
+      outer_reply_read = @reply_read # a job_parser that uses this client runs an _exchange inside ours
+      @reply_read = false # set by parse_response once the whole reply is off the socket
+      unless connection
+        _raise_not_connected! if @closed
+        raise Errno::ENOTCONN # _attempt reconnects and restores the tubes first
+      end
+
+      command = command.dup.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
+      read_window = _read_window(command) # checked before anything is written
+      _write(command.to_s + "\r\n", command)
+      @read_deadline = _deadline_in(read_window) # bounds the status line and the body read in parse_response
+      res = _read_line(command)
+      response = parse_response(command, res)
+      completed = true
+      response
+    rescue UnexpectedResponse => ex
+      # an error status line is the whole reply, so the stream is still in step
+      completed = !ex.is_a?(ExpectedCrlfError)
+      raise
+    ensure
+      # Interrupted between the write and the end of the reply (Timeout, Interrupt, an I/O
+      # error): the rest of the reply may still arrive, so drop the socket rather than let the
+      # next command read it. The next command reconnects. Once the whole reply has been read
+      # (a job_parser that raised, say) the stream is in step: keep the socket, and with it the
+      # jobs this connection has reserved.
+      _drop_connection unless completed || @reply_read
+      @reply_read = outer_reply_read
+    end
+
+    # Restores the watched and used tubes on a new connection. Sent with _exchange, not transmit:
+    # a failure here must go back to the _attempt that is reconnecting, which starts over.
     def _initialize_tubes
       if @tubes_watched != ['default']
         tubes_watched.each do |t|
-          transmit("watch #{t}", init: false)
+          _exchange("watch #{t}")
         end
 
-        transmit("ignore default", init: false)
+        _exchange("ignore default") unless tubes_watched.include?('default')
       end
 
-      transmit("use #{tube_used}", init: false) if @tube_used != 'default'
+      _exchange("use #{tube_used}") if @tube_used != 'default'
     end
 
-    # Wrapper method for capturing certain failures and retry the payload block
-    #
-    # @param [Proc] block The command to execute.
-    # @param [Integer] retry_interval The time to wait before the next retry
-    # @param [Integer] tries The maximum number of tries in draining mode
-    # @return [Object] Result of the block passed
-    #
-    def _with_retry(retry_interval: DEFAULT_RETRY_INTERVAL, init: true, tries: MAX_RETRIES, &block)
-      yield
-    rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
-      Errno::ECONNREFUSED => ex
-      _reconnect(ex, retry_interval)
-      _initialize_tubes if init
-      retry
+    # One attempt at a command, made holding the lock: reconnect (and restore the tubes) if the
+    # connection is gone, then send the command. Returns [true, response], or [false, seconds]
+    # to pause and try again. state carries the retry counts across attempts.
+    def _attempt(command, state)
+      if connection
+        state[:connecting] = false # another thread may have reconnected during our pause
+      else
+        _raise_not_connected! if @closed
+        unless state[:connecting] # a new reconnect, not the next try of one in progress
+          state[:reconnects] += 1
+          # something accepts connections but drops every command (a proxy with no backend):
+          # give up rather than reconnect in a tight loop
+          _raise_not_connected! if state[:reconnects] > MAX_RETRIES
+          state[:connecting] = true
+          state[:connect_tries] = MAX_RETRIES
+        end
+        begin
+          establish_connection
+        rescue SystemCallError, SocketError, IOError
+          state[:connect_tries] -= 1
+          _raise_not_connected! if state[:connect_tries].zero?
+          return [false, state[:retry_interval]]
+        end
+        state[:connecting] = false
+        if @closed # #close ran while connecting
+          _drop_connection
+          _raise_not_connected!
+        end
+        if state[:init]
+          begin
+            _initialize_tubes
+          rescue Exception
+            # never leave a half-restored connection watching the wrong tubes: drop it, so a
+            # lost connection reconnects and restores again below, and anything else makes the
+            # next command start over
+            _drop_connection
+            raise
+          end
+        end
+      end
+      [true, _exchange(command)]
+    rescue EOFError, Errno::ECONNRESET, Errno::EPIPE, Errno::ECONNREFUSED, Errno::ENOTCONN
+      _drop_connection
+      retry # reconnects at once, still holding the lock
     rescue Beaneater::DrainingError
-      tries -= 1
-      if tries.zero?
-        close
+      state[:draining_tries] -= 1
+      if state[:draining_tries].zero?
+        _drop_connection
         raise
       end
-      sleep(retry_interval)
-      retry
+      [false, state[:retry_interval]]
     end
 
-    # Tries to re-establish connection to the beanstalkd
-    #
-    # @param [Exception] original_exception The exception caused the retry
-    # @param [Integer] retry_interval The time to wait before the next reconnect
-    # @param [Integer] tries The maximum number of attempts to reconnect
-    def _reconnect(original_exception, retry_interval, tries=MAX_RETRIES)
-      close
-      establish_connection
-    rescue Errno::ECONNREFUSED
-      tries -= 1
-      if tries.zero?
-        _raise_not_connected!
-      end
-      sleep(retry_interval || DEFAULT_RETRY_INTERVAL)
-      retry
+    # Closes the socket without marking the connection closed, so the next command reconnects.
+    # Only #close is final.
+    def _drop_connection
+      socket, @connection = @connection, nil
+      @read_buffer = String.new
+      socket.close if socket
     end
 
     # The host provided by BEANSTALKD_URL environment variable, if available.
@@ -241,6 +404,99 @@ class Beaneater
       ENV['BEANSTALKD_URL'].respond_to?(:length) && ENV['BEANSTALKD_URL'].length > 0 && ENV['BEANSTALKD_URL'].strip
     end
 
+    # Converts one value of a stats or list response to its Ruby type:
+    # quoted values and text fields stay Strings, whole numbers become
+    # Integers, decimals Floats, and true/false Booleans.
+    def _stat_value(value, text)
+      if value.length >= 2 && value.start_with?('"') && value.end_with?('"')
+        value[1..-2]
+      elsif text
+        value
+      elsif value =~ /\A-?\d+\z/
+        value.to_i
+      elsif value =~ /\A-?\d+\.\d+\z/
+        value.to_f
+      elsif value == 'true' || value == 'false'
+        value == 'true'
+      else
+        value
+      end
+    end
+
+    # Writes data to the socket, waiting at most write_timeout seconds.
+    #
+    # Ruby waits on its sockets itself, so the socket-level SO_SNDTIMEO and
+    # SO_RCVTIMEO options have no effect; the deadlines are kept here instead.
+    def _write(data, cmd)
+      deadline = _deadline_in(config.write_timeout && Float(config.write_timeout))
+      until data.empty?
+        written = connection.write_nonblock(data, exception: false)
+        if written == :wait_writable
+          _wait_for(:write, deadline, cmd)
+        else
+          data = data.byteslice(written..-1)
+        end
+      end
+    end
+
+    # Returns how many seconds the reply to cmd may take once it is written, or
+    # nil to wait without limit. reserve-with-timeout gets its own timeout on top
+    # of read_timeout; a plain reserve blocks until a job arrives, by design.
+    # Timeouts may be given as numeric strings (from ENV, say).
+    def _read_window(cmd)
+      return nil if config.read_timeout.nil? || cmd =~ /\Areserve\s*\z/
+      extra = cmd =~ /\Areserve-with-timeout (\d+)/ ? $1.to_i : 0
+      Float(config.read_timeout) + extra
+    end
+
+    # Reads one reply line (the status line), without its line ending.
+    def _read_line(cmd)
+      until (eol = @read_buffer.index("\n"))
+        _fill_read_buffer(cmd)
+      end
+      @read_buffer.slice!(0, eol + 1)
+    end
+
+    # Reads exactly size bytes of a reply body.
+    def _read_bytes(size, cmd)
+      _fill_read_buffer(cmd) while @read_buffer.bytesize < size
+      @read_buffer.slice!(0, size)
+    end
+
+    # Waits until the socket has data (up to the reply's deadline) and appends
+    # what is available to the read buffer.
+    def _fill_read_buffer(cmd)
+      _wait_for(:read, @read_deadline, cmd)
+      @read_buffer << connection.readpartial(16_384)
+    end
+
+    # Waits for the socket to become readable or writable before deadline.
+    # On timeout the connection is dropped (the next command reconnects): a late
+    # reply would otherwise be read as the answer to the next command.
+    #
+    # @raise [Beaneater::SocketTimeout] The deadline passed.
+    def _wait_for(direction, deadline, cmd)
+      # Past the deadline this still polls once (timeout 0): data that is already there is
+      # not a timeout.
+      remaining = deadline && [deadline - _now, 0].max
+      # IO#wait_* rather than IO.select: a thread waiting here is woken by end-of-file when
+      # #close shuts the socket down (its command then raises NotConnected).
+      ready = direction == :read ? connection.wait_readable(remaining) : connection.wait_writable(remaining)
+      return if ready
+
+      _drop_connection
+      action = direction == :read ? 'waiting for the reply to' : 'writing'
+      raise SocketTimeout, "Timed out #{action} '#{cmd.to_s.split("\r\n").first}' on beanstalk '#{@host}:#{@port}'; the next command reconnects"
+    end
+
+    def _deadline_in(seconds)
+      seconds && _now + seconds
+    end
+
+    def _now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+
     # Raises an error to be triggered when the connection has failed
     # @raise [Beaneater::NotConnected] Beanstalkd is no longer connected
     def _raise_not_connected!

data/lib/beaneater/errors.rb

@@ -3,6 +3,20 @@
 class Beaneater
   # Raises when a beanstalkd instance is no longer accessible.
   class NotConnected < RuntimeError; end
+  # Raises when beanstalkd does not answer within the configured read_timeout,
+  # or a command cannot be written within write_timeout. The connection is
+  # closed, since a late reply would otherwise be read as the answer to the
+  # next command. Not to be confused with {TimedOutError}, which is
+  # beanstalkd's own TIMED_OUT reply to reserve-with-timeout.
+  class SocketTimeout < NotConnected; end
+  # Added to a copy of the exception a job_parser raises for a job just reserved. The copy
+  # keeps the exception's class (JSON::ParserError, say), message, backtrace and cause; #job_id
+  # names the job, still reserved by this connection, so the caller can bury, release or
+  # delete it.
+  module JobParseFailure
+    # @return [String] id of the reserved job whose body the parser rejected
+    attr_accessor :job_id
+  end
   # Raises when the tube name specified is invalid.
   class InvalidTubeName < RuntimeError; end
   # Raises when a job has not been reserved properly.