tuber 0.5.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5bf9725eaee223b9a587d41a70472161997c0792a7e3ad7917488e8e5dbacb62
-  data.tar.gz: 4759d84c9fae6eda87ed77f26817b6eae8d88fb2d80addf98723530c99551f7b
+  metadata.gz: 28e2397d8368c54a6711fa1db82b5c54ea649a4113e4da808b9dbc6db9cec1cb
+  data.tar.gz: dee769c5e37ff64726e1563150af3a25e2dc3f65405844f58b76bafbf4f7148f
 SHA512:
-  metadata.gz: 712f68857c4fcbedeaee298e94499abc9134337b1cd30de8f32c6a77fc8e640062bd6b1f0ba45122031d79a3c85cba7622c228f2d4447427e0d90c95668c6c47
-  data.tar.gz: 6bc153d12ffc33d3dcdd772ea31e8b2675a672779ace7812748bbd00c64346382bfea7c7d843ed0a24e610117f638f81445e0696dcfffa11633d93cc275cccd4
+  metadata.gz: 48a99c28b12b3f83cd1f24f743fabda6f06a97802c3e2633bb805f7309a3574508d45b3ba7c2ce1cfd7842b1f92159115e19f7331362c79d662c687f0b60af75
+  data.tar.gz: 60fa0ea0b9d9440ba798bf953f640f9a6c654bf3988bdca58bc51e037bfa347fefc606964c81a236335467cd5441ce4344ef9382a725e2920d69d8a871ba996f

data/CHANGELOG.md

@@ -4,6 +4,28 @@ The tuber gem is a fork of [beaneater](https://github.com/beanstalkd/beaneater),
 beaneater 1.1.4. Entries from 1.1.4 down are beaneater's history, kept for reference.
 (0.0.1 was an empty placeholder published to claim the gem name.)
 
+## 0.6.0 (September 16 2026)
+
+Reconnection release. The gem retried reconnects but not connects, so a
+consumer that caught `Tuber::NotConnected` and built a new client — the obvious
+thing to write — never reached the retrying path, and a server restart of a few
+seconds was enough to kill it. `reconnect!` makes the existing healing
+reachable, and `connect_retries` gives a cold start a window wider than the
+outage. Existing behaviour is unchanged until `connect_retries` is raised, with
+two exceptions noted below: `NotConnected` messages gained a suffix (the
+historical text remains a strict prefix), and a connect error other than
+`ECONNREFUSED` is now retried and surfaced as `NotConnected` rather than
+escaping as a raw `Errno`.
+
+* Add `Tuber#reconnect!` / `Connection#reconnect!`: re-establishes the connection and replays its tube state (watched tubes with their weights, the used tube, the reserve mode). This healing only ever ran when the connection itself noticed a dropped socket, so a consumer that caught `Tuber::NotConnected` had no way to reach it and had to build a new client — which starts with no tube state and has to re-watch and re-weight every tube by hand.
+* Add `connect_retries` and `connect_retry_interval` configuration. The initial connect took exactly one attempt with no retry, so a cold start during a server restart — a web boot, a fresh `Tuber.new`, a daemon rebuilding its client after an error — failed where an established connection would have healed. Defaults to the historical single attempt; raising it also lifts the reconnect budget of connections already established, so one knob covers both.
+* Retry a connect that fails for any reason the server might be unreachable (`ETIMEDOUT`, `EHOSTUNREACH`, `ENETUNREACH`, DNS failure, connect timeout), not only `ECONNREFUSED`. Those previously escaped as a raw `Errno` from the middle of a reconnect, a class no consumer thinks to rescue.
+* `Tuber::NotConnected` now names the underlying error in its message and keeps it as the exception's `cause`. A bare `rescue` collapsed every connect failure into "Connection to beanstalk '...' is closed!", hiding whether the server refused, timed out, or was never resolved — exactly what you need during an outage.
+* `Connection#close` no longer raises if the socket cannot be closed cleanly; the connection is gone either way and `reconnect!` must not fail on the way to replacing it.
+* Fix a reconnect silently dropping per-tube weights and the connection's reserve mode ([#1](https://github.com/tuberq/tuber-gem/issues/1)). Both are per-connection server state that a fresh socket resets, so a consumer that healed across a brief server restart kept working but quietly fell back to FIFO. `Connection` now remembers the weight each tube was watched with and the mode set by `Tubes#reserve_mode`, and replays them (watch, then `ignore default`, then `reserve-mode`) when re-initialising tubes.
+* Fix `_initialize_tubes` sending `ignore default` after a reconnect even when `default` was one of the watched tubes, which dropped it from the watch list.
+* `Connection#add_to_watched` now takes an optional weight as a second argument, and `Connection#tube_weights` / `Connection#reserve_mode` expose the replayed state.
+
 ## 0.5.1 (August 13 2026)
 
 * Fix empty keyword arguments forwarded through `Tube#transmit`, `Tubes#transmit` and `Jobs#transmit` arriving as an extra positional `{}` on Ruby 2.7

data/Gemfile

@@ -6,7 +6,9 @@ source 'https://rubygems.org'
 gemspec
 
 group :development do
-  gem 'redcarpet', '~> 1'
+  # 3.5.1 fixes CVE-2020-26298 (XSS when processing quotes). The old '~> 1'
+  # pin was inherited from beaneater and capped us below the fix.
+  gem 'redcarpet', '~> 3.6', '>= 3.5.1'
   gem 'github-markup'
   gem 'yard'
 end

data/README.md

@@ -72,6 +72,8 @@ Tuber.configure do |config|
   # config.resolv_timeout      = nil
   # config.read_timeout        = nil
   # config.write_timeout       = nil
+  # config.connect_retries        = 0
+  # config.connect_retry_interval = 1
 end
 ```
 
@@ -90,6 +92,47 @@ The above options are all defaults, so only include a configuration block if you
 @tuber.close
 ```
 
+#### Surviving a server restart
+
+An established connection heals itself: when a command notices the socket has
+died, the connection reconnects, replays its tube state, and re-sends the
+command (except for the non-idempotent verbs — `put`, `delete`, `release`,
+`bury`, `touch`, `kick` — where a blind re-send could duplicate a job or act on
+a reservation that died with the old socket, so the original error is raised
+for the caller to decide).
+
+That healing gets three connect attempts a second apart. A server restart
+typically takes longer, so give the client a budget that outlasts it:
+
+```ruby
+Tuber.configure do |config|
+  config.connect_retries        = 15   # extra attempts, 0 = a single attempt
+  config.connect_retry_interval = 1    # seconds between them
+end
+```
+
+`connect_retries` covers both a cold start (`Tuber.new` while the server is
+down) and reconnects on a client you already hold. It defaults to `0` — a
+single attempt — which is the historical behaviour.
+
+When a command does surface `Tuber::NotConnected`, heal the client you have
+rather than building a new one:
+
+```ruby
+begin
+  job = @tuber.tubes.reserve
+rescue Tuber::NotConnected
+  @tuber.reconnect!   # watched tubes, their weights, the used tube and the
+  retry               # reserve mode all come back with it
+end
+```
+
+A fresh `Tuber.new` starts with no tube state, so a worker that reconnects that
+way has to re-watch and re-weight every tube by hand — and will silently fall
+back to FIFO if it forgets the reserve mode. `reconnect!` replays all of it.
+It takes the same budget as the config, overridable per call:
+`@tuber.reconnect!(tries: 30, retry_interval: 2)`.
+
 ### Tubes
 
 Tubes are named work queues. Jobs are `put` into the used tube and `reserve`d from watched tubes. Each tube has a _ready_, _delayed_, and _buried_ queue.
@@ -270,6 +313,10 @@ job = @tuber.tubes.reserve  # batch-jobs selected 6x as often as email
 
 Tubes default to weight 1. Switch back with `reserve_mode(:fifo)`.
 
+Weights and reserve mode are per-connection server state. The client remembers
+both and replays them if the connection has to be re-established, so a worker
+that heals across a server blip keeps reserving by weight.
+
 ### Batch Reserve (Tuber only)
 
 Reserve multiple jobs atomically in a single call:
@@ -342,6 +389,10 @@ Tuber.configure do |config|
   config.job_parser          = lambda { |body| body }
   config.job_serializer      = lambda { |body| body }
   config.tuber_url           = 'localhost:11300'
+
+  # Connect attempts; see "Surviving a server restart" above.
+  config.connect_retries        = 0
+  config.connect_retry_interval = 1
 end
 ```
 

data/lib/tuber/configuration.rb

@@ -15,6 +15,21 @@ class Tuber
     attr_accessor :read_timeout        # socket read timeout in seconds
     attr_accessor :write_timeout       # socket write timeout in seconds
 
+    # How hard a client tries to connect — "how long should this ride out a
+    # server restart", in one knob.
+    #
+    # An established connection has always retried when it notices a dropped
+    # socket (Tuber::Connection::MAX_RETRIES attempts), but the *initial*
+    # connect got exactly one: a cold start during an outage — a web boot, a
+    # fresh Tuber.new, a daemon rebuilding its client after an error — failed
+    # where a held connection would have healed.
+    #
+    # Defaults to that single attempt, so nothing changes until you raise it.
+    # Raising it covers the initial connect and lifts the reconnect budget of
+    # connections already established along with it.
+    attr_accessor :connect_retries          # extra connect attempts, 0 = one shot
+    attr_accessor :connect_retry_interval   # seconds between connect attempts
+
     def initialize
       @default_put_delay   = 0
       @default_put_pri     = 65536
@@ -26,6 +41,9 @@ class Tuber
       @resolv_timeout      = nil
       @read_timeout        = nil
       @write_timeout       = nil
+      @connect_retries     = 0
+      # Matches Tuber::Connection::DEFAULT_RETRY_INTERVAL, which is not loaded yet.
+      @connect_retry_interval = 1
     end
   end # Configuration
 end # Tuber

data/lib/tuber/connection.rb

@@ -33,7 +33,17 @@ class Tuber
     #   @returns [Array<String>] returns currently watched tube names
     # @!attribute tube_used
     #   @returns [String] returns currently used tube name
-    attr_accessor :tubes_watched, :tube_used
+    # @!attribute reserve_mode
+    #   @returns [String, Symbol, nil] the reserve mode set on this connection,
+    #     nil when it was never set (the server default is fifo)
+    attr_accessor :tube_used, :reserve_mode
+    attr_reader :tubes_watched
+
+    # @!attribute tube_weights
+    #   @returns [Hash{String => Integer}] weight last watched with, per tube.
+    #     Tubes watched without a weight are absent: the server resets a tube's
+    #     weight to 1 on a plain `watch`, so "no weight" is state worth keeping.
+    attr_reader :tube_weights
 
     # Default port value for beanstalk connection
     DEFAULT_PORT = 11300
@@ -49,6 +59,25 @@ class Tuber
     # keeps the transparent retry.
     NON_IDEMPOTENT_COMMANDS = %w[put delete delete-batch release bury touch touch-all kick kick-job].freeze
 
+    # Errno names that mean "the server is not reachable right now" during a
+    # connect attempt. Every one of them is worth another try: a host coming
+    # back from a restart answers EHOSTUNREACH or ETIMEDOUT as readily as
+    # ECONNREFUSED, and a raw Errno escaping the connection is a class no
+    # consumer thinks to rescue.
+    CONNECT_ERROR_NAMES = %w[
+      ECONNREFUSED ECONNRESET ECONNABORTED ETIMEDOUT
+      EHOSTUNREACH EHOSTDOWN ENETUNREACH ENETDOWN EPIPE
+    ].freeze
+
+    # The resolved exception classes, skipping any Errno the platform lacks.
+    # SocketError covers a name that will not resolve (a DNS server restarting
+    # alongside the queue), IO::TimeoutError a connect_timeout on Ruby >= 3.2.
+    CONNECT_ERRORS = (
+      CONNECT_ERROR_NAMES.map { |name| Errno.const_get(name) if Errno.const_defined?(name) }.compact +
+      [SocketError] +
+      (defined?(IO::TimeoutError) ? [IO::TimeoutError] : [])
+    ).freeze
+
     # Initializes new connection.
     #
     # @param [String] address beanstalkd instance address.
@@ -61,15 +90,23 @@ class Tuber
     #   @b.connection.host # => '127.0.0.1'
     #   @b.connection.port # => '11300'
     #
+    # @raise [Tuber::NotConnected] Could not connect. The underlying error is
+    #   named in the message and kept as the exception's +cause+.
+    #
     def initialize(address)
       @address = address || _host_from_env || Tuber.configuration.tuber_url
       @mutex = Mutex.new
       @tube_used = 'default'
       @tubes_watched = ['default']
-
-      establish_connection
-    rescue
-      _raise_not_connected!
+      @tube_weights = {}
+      @reserve_mode = nil
+
+      _connect(tries: config.connect_retries.to_i + 1,
+               retry_interval: config.connect_retry_interval)
+    rescue Tuber::NotConnected
+      raise # already carries its cause; do not wrap it twice
+    rescue => ex
+      _raise_not_connected!(ex)
     end
 
     # Send commands to beanstalkd server via connection.
@@ -170,11 +207,50 @@ class Tuber
     #
     def close
       if @connection
-        @connection.close
+        begin
+          @connection.close
+        rescue StandardError
+          # A socket whose peer has already vanished can fail to close
+          # cleanly. It is gone either way, and #reconnect! must not raise
+          # on the way to replacing it.
+        end
         @connection = nil
       end
     end
 
+    # Re-establishes this connection and replays its tube state onto the new
+    # socket: watched tubes with the weights they were watched with, the used
+    # tube, and the reserve mode. Safe to call whether the current socket is
+    # healthy, dropped, or already closed.
+    #
+    # This is the same healing the connection performs internally when a
+    # command notices the socket has died, exposed for consumers that hold a
+    # connection across an outage. Prefer it to building a new Tuber: a fresh
+    # Connection makes a single connect attempt by default (see
+    # Tuber::Configuration#connect_retries) and starts with no tube state, so
+    # the caller has to re-watch and re-weight everything by hand.
+    #
+    # @param [Integer, nil] tries Maximum number of connect attempts, nil for
+    #   the configured default (see Tuber::Configuration#connect_retries)
+    # @param [Numeric, nil] retry_interval Seconds to wait between attempts,
+    #   nil for the configured default
+    # @return [Tuber::Connection] self
+    # @raise [Tuber::NotConnected] Every connect attempt failed
+    # @example
+    #   begin
+    #     job = @conn.transmit("reserve")
+    #   rescue Tuber::NotConnected
+    #     @conn.reconnect!   # watches, weights and reserve mode come back with it
+    #     retry
+    #   end
+    #
+    def reconnect!(tries: nil, retry_interval: nil)
+      _reconnect(tries: tries || _connect_tries,
+                 retry_interval: retry_interval || config.connect_retry_interval)
+      _initialize_tubes
+      self
+    end
+
     # Returns string representation of job.
     #
     # @example
@@ -185,15 +261,34 @@ class Tuber
     end
     alias :inspect :to_s
 
-    def add_to_watched(tube_name)
+    # Records a tube as watched, along with the weight it was watched with.
+    #
+    # @param [String] tube_name Name of the tube now being watched
+    # @param [Integer, nil] weight Weight passed to `watch`, nil for a plain watch
+    def add_to_watched(tube_name, weight = nil)
       @tubes_watched << tube_name
-      @tubes_watched.uniq
+      @tubes_watched.uniq!
+      if weight
+        @tube_weights[tube_name] = weight
+      else
+        @tube_weights.delete(tube_name)
+      end
+      @tubes_watched
     end
 
     def remove_from_watched(tube_name)
+      @tube_weights.delete(tube_name)
       @tubes_watched.delete(tube_name)
     end
 
+    # Replaces the watched tube list, dropping weights for tubes that are no
+    # longer watched.
+    def tubes_watched=(tube_names)
+      @tubes_watched = tube_names
+      @tube_weights.keep_if { |name, _| @tubes_watched.include?(name) }
+      @tubes_watched
+    end
+
     protected
 
     # Establish a connection based on beanstalk address.
@@ -278,16 +373,24 @@ class Tuber
 
     private
 
+    # Replays this connection's tube state onto a freshly established socket.
+    # Weights and reserve mode are per-connection server state that a new socket
+    # resets, so a reconnect that only re-watched the tube names would silently
+    # drop a weighted consumer back to FIFO. Order matters: watch, then drop
+    # default, then set the mode.
     def _initialize_tubes
-      if @tubes_watched != ['default']
+      if @tubes_watched != ['default'] || @tube_weights.any?
         tubes_watched.each do |t|
-          transmit("watch #{t}", init: false)
+          weight = @tube_weights[t]
+          transmit(weight ? "watch #{t} #{weight}" : "watch #{t}", init: false)
         end
 
-        transmit("ignore default", init: false)
+        transmit("ignore default", init: false) unless @tubes_watched.include?('default')
       end
 
       transmit("use #{tube_used}", init: false) if @tube_used != 'default'
+
+      transmit("reserve-mode #{@reserve_mode}", init: false) if @reserve_mode && @reserve_mode.to_s != 'fifo'
     end
 
     # Wrapper method for capturing certain failures and retry the payload block
@@ -304,7 +407,7 @@ class Tuber
       yield
     rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
       Errno::ECONNREFUSED => ex
-      _reconnect(ex, retry_interval)
+      _reconnect(tries: _connect_tries, retry_interval: retry_interval)
       _initialize_tubes if init
       raise ex unless retransmit
       retry
@@ -318,19 +421,37 @@ class Tuber
       retry
     end
 
-    # Tries to re-establish connection to the beanstalkd
+    # Connect attempts to allow when healing a connection that was already
+    # established. Never fewer than MAX_RETRIES, so the transparent retry
+    # behaves exactly as it always has by default; raising connect_retries
+    # lifts this with it, making one knob answer "how long should a client
+    # ride out a server restart" for cold starts and reconnects alike.
+    #
+    # @return [Integer] maximum number of connect attempts
+    def _connect_tries
+      [MAX_RETRIES, config.connect_retries.to_i + 1].max
+    end
+
+    # Drops the current socket and connects again.
     #
-    # @param [Exception] original_exception The exception caused the retry
-    # @param [Integer] retry_interval The time to wait before the next reconnect
     # @param [Integer] tries The maximum number of attempts to reconnect
-    def _reconnect(original_exception, retry_interval, tries=MAX_RETRIES)
+    # @param [Numeric] retry_interval The time to wait before the next attempt
+    # @raise [Tuber::NotConnected] Every attempt failed
+    def _reconnect(tries: MAX_RETRIES, retry_interval: DEFAULT_RETRY_INTERVAL)
       close
+      _connect(tries: tries, retry_interval: retry_interval)
+    end
+
+    # Connects, retrying while the server is refusing or unreachable.
+    #
+    # @param [Integer] tries The maximum number of connect attempts
+    # @param [Numeric] retry_interval The time to wait between attempts
+    # @raise [Tuber::NotConnected] Every attempt failed
+    def _connect(tries: MAX_RETRIES, retry_interval: DEFAULT_RETRY_INTERVAL)
       establish_connection
-    rescue Errno::ECONNREFUSED
+    rescue *CONNECT_ERRORS => ex
       tries -= 1
-      if tries.zero?
-        _raise_not_connected!
-      end
+      _raise_not_connected!(ex) if tries <= 0
       sleep(retry_interval || DEFAULT_RETRY_INTERVAL)
       retry
     end
@@ -356,10 +477,23 @@ class Tuber
       [sec, usec].pack('l_l_')
     end
 
-    # Raises an error to be triggered when the connection has failed
+    # Raises an error to be triggered when the connection has failed.
+    #
+    # The underlying error is named in the message as well as kept as the
+    # exception's +cause+. A log line reading only "Connection to beanstalk
+    # '...' is closed!" hides whether the server refused, timed out, or never
+    # resolved — which is exactly what you want to know during an outage.
+    #
+    # Only the two callers that are genuinely inside a rescue pass a cause.
+    # Defaulting to +$!+ would be wrong: a command issued from inside someone
+    # else's rescue block would name whatever they were handling.
+    #
+    # @param [Exception, nil] cause The error that stopped the connect
     # @raise [Tuber::NotConnected] Beanstalkd is no longer connected
-    def _raise_not_connected!
-      raise Tuber::NotConnected, "Connection to beanstalk '#{@host}:#{@port}' is closed!"
+    def _raise_not_connected!(cause = nil)
+      message = "Connection to beanstalk '#{@host}:#{@port}' is closed!"
+      message += " (#{cause.class}: #{cause.message})" if cause && !cause.is_a?(Tuber::NotConnected)
+      raise Tuber::NotConnected, message
     end
 
   end # Connection

data/lib/tuber/tube/collection.rb

@@ -103,7 +103,11 @@ class Tuber
     #
     # @api public
     def reserve_mode(mode)
-      transmit("reserve-mode #{mode}")
+      res = transmit("reserve-mode #{mode}")
+      # Remembered so a reconnect can replay it: reserve mode is per-connection
+      # server state, and a fresh socket starts back at fifo.
+      client.connection.reserve_mode = mode
+      res
     end
 
     # Reserves a specific job by its ID.
@@ -202,7 +206,7 @@ class Tuber
       names.each do |t|
         cmd = weight ? "watch #{t} #{weight}" : "watch #{t}"
         transmit cmd
-        client.connection.add_to_watched(t)
+        client.connection.add_to_watched(t, weight)
       end
     rescue BadFormatError => ex
       raise InvalidTubeName, "Tube in '#{ex.cmd}' is invalid!"

data/lib/tuber/version.rb

@@ -2,5 +2,5 @@
 
 class Tuber
   # Current version of gem.
-  VERSION = "0.5.1"
+  VERSION = "0.6.0"
 end

data/lib/tuber.rb

@@ -71,6 +71,27 @@ class Tuber
     connection.transmit("undrain")
   end
 
+  # Re-establishes the connection and replays its tube state: watched tubes
+  # with their weights, the used tube, and the reserve mode.
+  #
+  # Use this rather than building a new Tuber when a connection drops — a new
+  # client starts with no tube state and, by default, gets a single connect
+  # attempt, so it loses a race a reconnect would have won.
+  #
+  # @param [Integer, nil] tries Maximum number of connect attempts, nil for the
+  #   configured default (see Tuber::Configuration#connect_retries)
+  # @param [Numeric, nil] retry_interval Seconds between attempts, nil for the
+  #   configured default
+  # @return [Tuber] self
+  # @raise [Tuber::NotConnected] Every connect attempt failed
+  # @example
+  #  @tuber_instance.reconnect!
+  #
+  def reconnect!(tries: nil, retry_interval: nil)
+    connection.reconnect!(tries: tries, retry_interval: retry_interval)
+    self
+  end
+
   # Closes the related connection
   #
   # @example

data/test/connection_test.rb

@@ -228,6 +228,131 @@ describe Tuber::Connection do
       # non-retransmitted command (delete/put) now surfaces as a failure.
       $called = true
     end
+
+    it "replays tube weights and reserve mode after reconnect" do
+      require_tuber!
+      client = Tuber.new(tuber_address)
+      client.tubes.watch("reconn_heavy", weight: 8)
+      client.tubes.watch("reconn_light", weight: 1)
+      client.tubes.ignore("default")
+      client.tubes.reserve_mode(:weighted)
+
+      $replayed = []
+      $eof_pending = true
+      TCPSocket.prepend Module.new {
+        def write(data)
+          $replayed << data.to_s if $replayed
+          super
+        end
+
+        def readline
+          if $eof_pending
+            $eof_pending = false
+            raise EOFError
+          end
+
+          super
+        end
+      }
+
+      client.connection.transmit("list-tubes-watched")
+      sent = $replayed
+
+      assert_includes sent, "watch reconn_heavy 8\r\n"
+      assert_includes sent, "watch reconn_light 1\r\n"
+      assert_includes sent, "reserve-mode weighted\r\n"
+      # Ordering matters: watch, then drop default, then set the mode.
+      assert sent.index("watch reconn_heavy 8\r\n") < sent.index("ignore default\r\n"),
+        "expected the watches to be replayed before 'ignore default'"
+      assert sent.index("ignore default\r\n") < sent.index("reserve-mode weighted\r\n"),
+        "expected 'reserve-mode' to be replayed after the tubes are watched"
+    ensure
+      # Leave the prepended module inert for the rest of the suite.
+      $eof_pending = false
+      $replayed = nil
+      client.close if client
+    end
+
+    it "keeps watching default across a reconnect when default is watched" do
+      client = Tuber.new(tuber_address)
+      client.tubes.watch("reconn_extra")
+
+      $replayed = []
+      $eof_pending = true
+      TCPSocket.prepend Module.new {
+        def write(data)
+          $replayed << data.to_s if $replayed
+          super
+        end
+
+        def readline
+          if $eof_pending
+            $eof_pending = false
+            raise EOFError
+          end
+
+          super
+        end
+      }
+
+      client.connection.transmit("list-tubes-watched")
+      sent = $replayed
+
+      assert_includes sent, "watch reconn_extra\r\n"
+      refute_includes sent, "ignore default\r\n"
+      assert_equal %w[default reconn_extra].sort,
+        client.tubes.watched.map(&:name).sort
+    ensure
+      $eof_pending = false
+      $replayed = nil
+      client.close if client
+    end
+
+    it "still reserves by weight after a reconnect" do
+      require_tuber!
+      consumer = Tuber.new(tuber_address)
+      consumer.tubes.watch("reconn_heavy", weight: 8)
+      consumer.tubes.watch("reconn_light", weight: 1)
+      consumer.tubes.ignore("default")
+      consumer.tubes.reserve_mode(:weighted)
+
+      producer = Tuber.new(tuber_address)
+      100.times do |i|
+        producer.tubes["reconn_heavy"].put("h#{i}")
+        producer.tubes["reconn_light"].put("l#{i}")
+      end
+
+      $eof_pending = true
+      TCPSocket.prepend Module.new {
+        def readline
+          if $eof_pending
+            $eof_pending = false
+            raise EOFError
+          end
+
+          super
+        end
+      }
+
+      counts = Hash.new(0)
+      60.times do
+        job = consumer.tubes.reserve(2)
+        counts[job.body[0]] += 1
+        job.delete
+      end
+
+      # 8:1 over interleaved puts is ~53/7; FIFO would be ~30/30.
+      assert counts["h"] > 3 * counts["l"],
+        "expected weighted reserve after reconnect, got heavy=#{counts["h"]} light=#{counts["l"]}"
+    ensure
+      $eof_pending = false
+      if producer
+        producer.connection.transmit("flush-tube reconn_heavy") rescue nil
+        producer.connection.transmit("flush-tube reconn_light") rescue nil
+        producer.close
+      end
+      consumer.close if consumer
+    end
   end # transmit
 
   describe 'for idempotent insert response' do
@@ -251,6 +376,217 @@ describe Tuber::Connection do
     end
   end # idempotent insert response
 
+  describe 'for #reconnect!' do
+    before do
+      @host = tuber_address
+      @bc = Tuber::Connection.new(@host)
+    end
+
+    after do
+      @bc.close rescue nil
+    end
+
+    it "re-establishes a connection that was closed" do
+      @bc.close
+      assert_nil @bc.connection
+
+      @bc.reconnect!
+
+      assert_kind_of TCPSocket, @bc.connection
+      assert_equal 'OK', @bc.transmit('stats')[:status]
+    end
+
+    it "replays watched tubes, weights and reserve mode" do
+      require_tuber!
+      client = Tuber.new(tuber_address)
+      client.tubes.watch("reconn_bang_heavy", weight: 8)
+      client.tubes.watch("reconn_bang_light", weight: 1)
+      client.tubes.ignore("default")
+      client.tubes.reserve_mode(:weighted)
+
+      $replayed = []
+      TCPSocket.prepend Module.new {
+        def write(data)
+          $replayed << data.to_s if $replayed
+          super
+        end
+      }
+
+      client.connection.reconnect!
+      sent = $replayed
+
+      assert_includes sent, "watch reconn_bang_heavy 8\r\n"
+      assert_includes sent, "watch reconn_bang_light 1\r\n"
+      assert_includes sent, "ignore default\r\n"
+      assert_includes sent, "reserve-mode weighted\r\n"
+      assert_equal %w[reconn_bang_heavy reconn_bang_light].sort,
+        client.tubes.watched.map(&:name).sort
+    ensure
+      # Leave the prepended module inert for the rest of the suite.
+      $replayed = nil
+      client.close if client
+    end
+
+    it "retries while the server is still refusing connections" do
+      # The outage shape: the socket is gone and the server will not accept
+      # for another second or two. A single connect attempt loses that race.
+      @bc.close
+      real_socket = TCPSocket.new(tuber_address, 11300)
+      TCPSocket.stubs(:new).
+        raises(Errno::ECONNREFUSED.new).then.
+        raises(Errno::ECONNREFUSED.new).then.
+        returns(real_socket)
+
+      @bc.reconnect!(retry_interval: 0.01)
+
+      assert_same real_socket, @bc.connection
+    end
+
+    it "retries a connect that fails for a reason other than ECONNREFUSED" do
+      # A remote host mid-restart answers EHOSTUNREACH or ETIMEDOUT as readily
+      # as ECONNREFUSED; those must not escape as a raw Errno either.
+      @bc.close
+      real_socket = TCPSocket.new(tuber_address, 11300)
+      TCPSocket.stubs(:new).
+        raises(Errno::EHOSTUNREACH.new).then.
+        returns(real_socket)
+
+      @bc.reconnect!(retry_interval: 0.01)
+
+      assert_same real_socket, @bc.connection
+    end
+
+    it "raises NotConnected naming the underlying error when the server stays down" do
+      @bc.close
+      TCPSocket.stubs(:new).raises(Errno::ECONNREFUSED.new)
+
+      err = assert_raises(Tuber::NotConnected) { @bc.reconnect!(retry_interval: 0.01) }
+      assert_match(/ECONNREFUSED/, err.message)
+      assert_kind_of Errno::ECONNREFUSED, err.cause
+    end
+
+    it "is reachable from the Tuber instance" do
+      client = Tuber.new(tuber_address)
+      client.tubes.watch("reconn_delegate")
+
+      client.reconnect!
+
+      assert_includes client.tubes.watched.map(&:name), "reconn_delegate"
+    ensure
+      client.close if client
+    end
+  end # reconnect!
+
+  describe 'for connect retries' do
+    after do
+      Tuber.configure do |config|
+        config.connect_retries        = 0
+        config.connect_retry_interval = Tuber::Connection::DEFAULT_RETRY_INTERVAL
+      end
+    end
+
+    it "should make a single connect attempt by default" do
+      TCPSocket.expects(:new).times(1).raises(Errno::ECONNREFUSED.new)
+
+      assert_raises(Tuber::NotConnected) { Tuber::Connection.new('localhost') }
+    end
+
+    it "should retry the initial connect when connect_retries is configured" do
+      real_socket = TCPSocket.new(tuber_address, 11300)
+      Tuber.configure do |config|
+        config.connect_retries        = 3
+        config.connect_retry_interval = 0.01
+      end
+
+      TCPSocket.stubs(:new).
+        raises(Errno::ECONNREFUSED.new).then.
+        raises(Errno::ECONNREFUSED.new).then.
+        returns(real_socket)
+
+      bc = Tuber::Connection.new(tuber_address)
+
+      assert_same real_socket, bc.connection
+    end
+
+    it "should give up after connect_retries and name the cause" do
+      Tuber.configure do |config|
+        config.connect_retries        = 2
+        config.connect_retry_interval = 0.01
+      end
+      TCPSocket.expects(:new).times(3).raises(Errno::ECONNREFUSED.new)
+
+      err = assert_raises(Tuber::NotConnected) { Tuber::Connection.new('localhost') }
+      assert_match(/ECONNREFUSED/, err.message)
+      assert_kind_of Errno::ECONNREFUSED, err.cause
+    end
+
+    it "should lift an established connection's reconnect budget too" do
+      # One knob for "how long should a client ride out a server restart":
+      # raising connect_retries has to cover the connection you already hold,
+      # not only a cold start.
+      bc = Tuber::Connection.new(tuber_address)
+      real_socket = TCPSocket.new(tuber_address, 11300)
+
+      Tuber.configure do |config|
+        config.connect_retries        = 5
+        config.connect_retry_interval = 0.01
+      end
+
+      # One dropped read, then four refused connects: past the built-in budget
+      # of MAX_RETRIES, inside the configured one.
+      $eof_pending = true
+      TCPSocket.prepend Module.new {
+        def readline
+          if $eof_pending
+            $eof_pending = false
+            raise EOFError
+          end
+
+          super
+        end
+      }
+      TCPSocket.stubs(:new).
+        raises(Errno::ECONNREFUSED.new).then.
+        raises(Errno::ECONNREFUSED.new).then.
+        raises(Errno::ECONNREFUSED.new).then.
+        raises(Errno::ECONNREFUSED.new).then.
+        returns(real_socket)
+
+      assert_equal 'OK', bc.transmit('stats')[:status]
+      assert_same real_socket, bc.connection
+    ensure
+      $eof_pending = false
+    end
+  end # connect retries
+
+  describe 'for NotConnected messages' do
+    it "should keep the historical message as a prefix" do
+      # Consumers match on this string. The cause is appended, never inserted.
+      TCPSocket.stubs(:new).raises(Errno::ECONNREFUSED.new)
+
+      err = assert_raises(Tuber::NotConnected) { Tuber::Connection.new('localhost:8544') }
+
+      assert err.message.start_with?("Connection to beanstalk 'localhost:8544' is closed!"),
+        "expected the historical message as a prefix, got #{err.message.inspect}"
+    end
+
+    it "should not blame an unrelated in-flight exception" do
+      # Commands are very often issued from inside someone else's rescue block,
+      # where $! is still set. The cause named has to be the one that stopped
+      # this connect, not whatever the caller happened to be handling.
+      bc = Tuber::Connection.new(tuber_address)
+      bc.close
+
+      err = begin
+        raise ArgumentError, "something else entirely"
+      rescue ArgumentError
+        assert_raises(Tuber::NotConnected) { bc.transmit('stats') }
+      end
+
+      assert_equal "Connection to beanstalk '#{bc.host}:#{bc.port}' is closed!", err.message
+    end
+  end # NotConnected messages
+
   describe 'for #close' do
     before do
       @host = 'localhost'

data/test/tubes_test.rb

@@ -128,13 +128,13 @@ describe Tuber::Tubes do
 
     it 'should send watch command with weight' do
       @connection.expects(:transmit).with("watch foo 4").returns({status: "WATCHING", id: "2"})
-      @connection.expects(:add_to_watched).with("foo")
+      @connection.expects(:add_to_watched).with("foo", 4)
       @tubes.watch('foo', weight: 4)
     end
 
     it 'should send watch command without weight by default' do
       @connection.expects(:transmit).with("watch foo").returns({status: "WATCHING", id: "2"})
-      @connection.expects(:add_to_watched).with("foo")
+      @connection.expects(:add_to_watched).with("foo", nil)
       @tubes.watch('foo')
     end
   end # watch with weight
@@ -149,13 +149,26 @@ describe Tuber::Tubes do
 
     it 'should send reserve-mode weighted command' do
       @connection.expects(:transmit).with("reserve-mode weighted").returns({status: "USING", id: "weighted"})
+      @connection.expects(:reserve_mode=).with(:weighted)
       @tubes.reserve_mode(:weighted)
     end
 
     it 'should send reserve-mode fifo command' do
       @connection.expects(:transmit).with("reserve-mode fifo").returns({status: "USING", id: "fifo"})
+      @connection.expects(:reserve_mode=).with(:fifo)
       @tubes.reserve_mode(:fifo)
     end
+
+    it 'should record the mode on the connection so a reconnect can replay it' do
+      require_tuber!
+      client = Tuber.new(tuber_address)
+      begin
+        client.tubes.reserve_mode(:weighted)
+        assert_equal :weighted, client.connection.reserve_mode
+      ensure
+        client.close
+      end
+    end
   end # reserve_mode
 
   describe "for #reserve" do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: tuber
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.6.0
 platform: ruby
 authors:
 - Dan Milne
@@ -97,7 +97,6 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
-- TODO
 - examples/demo.rb
 - lib/tuber.rb
 - lib/tuber/configuration.rb

data/TODO

@@ -1 +0,0 @@
-- Remove connection from pool if it's not responding and be able to add more connections and reattempt later