omq 0.15.5 → 0.16.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a3c8e182cc758212708988bb4c88a7d374c63f06053e3be829c3a98620e23450
-  data.tar.gz: da1bdea2c47ec1c0a8e5e15d0b4ea69c9af642cd7fc67eae4b0931209314eac3
+  metadata.gz: f74ea96d0fc94d40c2d5ac0dd815e112a8fc8af8f74d5b932799fb19981c58b5
+  data.tar.gz: f79ddfe0a3519f7011353c03eeb18f054e9c5384e3e67250db699179c86be96e
 SHA512:
-  metadata.gz: 6d56451299b8e3cfc500ea145826d8e1bc251edf814dc0e4fea6846428ff546bd3f7f66005e4c41b55d9b4b54c6e536f62619de6086311cdd538a16080e2802e
-  data.tar.gz: 1f5b9061e5eae37357c4d6e1dc828bac2778f68ba6a10072faecc4d049b9517e2044b66bc4771b1a16dc38717583e7689cb74b9e653b0df15171e0723bd1150a
+  metadata.gz: 6ef57ca7cfd08b2add9adc5dc501a1a587b29ddb534ca55b20d1bf8cebf3571538c79cfc7ed1e1d2cddf93b69993b9f674c951cf37d32be54c93550465f00dca
+  data.tar.gz: 5d847021b68bed1f6b93d128afdc455c2354c8dc11febf8a711624c5559d4d03786a041db2b0d797b4518a789971029f9db8b6ad94becf44ae807a14ae396cf2

data/CHANGELOG.md

@@ -1,5 +1,86 @@
 # Changelog
 
+## 0.16.1 — 2026-04-09
+
+### Changed
+
+- **Depend on `protocol-zmtp ~> 0.4`.** Picks up the batched
+  `Connection#write_messages` used by the work-stealing send pumps and
+  the zero-alloc frame-header path on the unencrypted hot send path.
+
+### Fixed
+
+- **PUB/XPUB/RADIO fan-out now honors `on_mute`.** Per-subscriber send queues
+  were hardcoded to `:block`, so a slow subscriber would back-pressure the
+  publisher despite PUB/XPUB/RADIO defaulting to `on_mute: :drop_newest`.
+  Fan-out now builds each subscriber's queue with the socket's `on_mute`
+  strategy — slow subscribers silently drop their own messages without
+  stalling the publisher or other subscribers.
+
+## 0.16.0 — 2026-04-09
+
+### Changed
+
+- **Consolidate connection lifecycle into `Engine::ConnectionLifecycle`.** One
+  object per connection owns the full arc: handshake → ready → closed. Replaces
+  the scattered callback pattern where `Engine`, `ConnectionSetup`, and
+  `#close_connections_at` each held partial responsibility for registration,
+  monitor emission, routing add/remove, and reconnect scheduling. Side-effect
+  order (`:handshake_succeeded` before `connection_added`, `connection_removed`
+  before `:disconnected`) is now encoded as sequential statements in two
+  methods instead of implicit across multiple files. Teardown is idempotent via
+  an explicit 4-state transition table — racing pumps can no longer
+  double-fire `:disconnected` or double-call `routing.connection_removed`.
+  `ConnectionSetup` is absorbed and removed. `ConnectionRecord` collapses away
+  — `@connections` now stores lifecycles directly.
+
+- **Consolidate socket-level state into `Engine::SocketLifecycle`.** Six ivars
+  (`@state`, `@peer_connected`, `@all_peers_gone`, `@reconnect_enabled`,
+  `@parent_task`, `@on_io_thread`) move into one cohesive object with an
+  explicit 4-state transition table (`:new → :open → :closing → :closed`).
+  `Engine#closed?`, `#peer_connected`, `#all_peers_gone`, `#parent_task`
+  remain as delegators — public API unchanged. Parallels
+  `ConnectionLifecycle` in naming and shape. Pure refactor, no behavior change.
+
+- **Revert to per-socket HWM with work-stealing send pumps.** One shared
+  bounded send queue per socket, drained by N per-connection send pumps
+  that race to dequeue. Slow peers' pumps simply stop pulling; fast peers
+  absorb the load. Strictly better PUSH semantics than libzmq's strict
+  per-pipe round-robin (a known footgun where one slow worker stalls the
+  whole pipeline). Removes `StagingQueue`, per-connection queue maps, the
+  double-drain race in `add_*`, the disconnect-prepend ordering pretense,
+  and the `@cycle` / next-connection machinery. See `DESIGN.md`
+  "Per-socket HWM (not per-connection)" for full reasoning.
+- **`RoundRobin` batch cap is now dual: 256 messages OR 512 KB**, whichever
+  hits first (previously 64 messages). The old cap was too aggressive for
+  large messages — with 64 KB payloads it forced a flush every ~4 MB,
+  capping multi-peer push_pull throughput at ~50 % of what the network
+  could handle. Dual cap lets large-message workloads batch ~8 messages
+  per cycle while small-message workloads still yield quickly enough to
+  keep other work-stealing pumps fair. push_pull +5–40 % across transports
+  and sizes; router_dealer +5–15 %.
+- **Send pumps batched under a single mutex.** RoundRobin, ConnSendPump
+  and Pair now drain batches through
+  `Protocol::ZMTP::Connection#write_messages`, collapsing N lock
+  acquire/release pairs into one per batch. The size==1 path still uses
+  `send_message` (write+flush in one lock) to avoid an extra round-trip
+  at low throughput. push_pull inproc +18–28 %, tcp/ipc flat to +17 %.
+
+### Fixed
+
+- **`disconnect(endpoint)` now emits `:disconnected`** on the monitor queue.
+  Previously silent because `close_connections_at` bypassed `connection_lost`.
+- **PUSH/PULL round-robin test.** Previously asserted strict 1-msg-per-peer
+  distribution — a libzmq-ism OMQ never promised — and was silently
+  "passing" with 0 assertions and a 10 s Async-block timeout that masked a
+  hang. New test verifies both peers receive nonzero load over TCP.
+
+### Benchmarks
+
+- Report throughput in bytes/s alongside msgs/s.
+- Regenerated `bench/README.md` PUSH/PULL and REQ/REP tables: push_pull
+  throughput up 5–40 %, req_rep round-trip latency down 5–15 %.
+
 ## 0.15.5 — 2026-04-08
 
 - **`max_message_size` now defaults to `nil` (unlimited)** — previous

data/lib/omq/engine/connection_lifecycle.rb

@@ -0,0 +1,148 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Owns the full arc of one connection: handshake → ready → closed.
+    #
+    # Centralizes the ordering of side effects (monitor events, routing
+    # registration, promise resolution, reconnect scheduling) so the
+    # sequence lives in one place instead of being scattered across
+    # Engine, ConnectionSetup, and close paths.
+    #
+    # State machine:
+    #
+    #   new ──┬── :handshaking ── :ready ── :closed
+    #         └── :ready ── :closed            (inproc fast path)
+    #
+    # #lost! and #close! are idempotent — the state guard ensures side
+    # effects run exactly once even if multiple pumps race to report a
+    # lost connection.
+    #
+    class ConnectionLifecycle
+      class InvalidTransition < RuntimeError; end
+
+      STATES = %i[new handshaking ready closed].freeze
+
+      TRANSITIONS = {
+        new:         %i[handshaking ready closed].freeze,
+        handshaking: %i[ready closed].freeze,
+        ready:       %i[closed].freeze,
+        closed:      [].freeze,
+      }.freeze
+
+
+      # @return [Protocol::ZMTP::Connection, Transport::Inproc::DirectPipe, nil]
+      attr_reader :conn
+
+      # @return [String, nil]
+      attr_reader :endpoint
+
+      # @return [Symbol] current state
+      attr_reader :state
+
+
+      # @param engine [Engine]
+      # @param endpoint [String, nil]
+      # @param done [Async::Promise, nil] resolved when connection is lost
+      #
+      def initialize(engine, endpoint: nil, done: nil)
+        @engine   = engine
+        @endpoint = endpoint
+        @done     = done
+        @state    = :new
+        @conn     = nil
+      end
+
+
+      # Performs the ZMTP handshake and transitions to :ready.
+      #
+      # @param io [#read, #write, #close]
+      # @param as_server [Boolean]
+      # @return [Protocol::ZMTP::Connection]
+      #
+      def handshake!(io, as_server:)
+        transition!(:handshaking)
+        conn = Protocol::ZMTP::Connection.new(
+          io,
+          socket_type:      @engine.socket_type.to_s,
+          identity:         @engine.options.identity,
+          as_server:        as_server,
+          mechanism:        @engine.options.mechanism&.dup,
+          max_message_size: @engine.options.max_message_size,
+        )
+        conn.handshake!
+        Heartbeat.start(Async::Task.current, conn, @engine.options, @engine.tasks)
+        ready!(conn)
+        @conn
+      rescue Protocol::ZMTP::Error, *CONNECTION_LOST => error
+        @engine.emit_monitor_event(:handshake_failed, endpoint: @endpoint, detail: { error: error })
+        conn&.close
+        transition!(:closed)
+        raise
+      end
+
+
+      # Registers an already-connected inproc pipe as :ready.
+      # No handshake — inproc DirectPipe bypasses ZMTP entirely.
+      #
+      # @param pipe [Transport::Inproc::DirectPipe]
+      #
+      def ready_direct!(pipe)
+        ready!(pipe)
+      end
+
+
+      # Transitions to :closed, running the full loss sequence:
+      # routing removal, monitor event, reconnect scheduling.
+      # Idempotent: a no-op if already :closed.
+      #
+      def lost!
+        tear_down!(reconnect: true)
+      end
+
+
+      # Transitions to :closed without scheduling a reconnect.
+      # Used by shutdown paths (Engine#close, #disconnect, #unbind).
+      # Idempotent.
+      #
+      def close!
+        tear_down!(reconnect: false)
+      end
+
+
+      private
+
+      def ready!(conn)
+        conn  = @engine.connection_wrapper.call(conn) if @engine.connection_wrapper
+        @conn = conn
+        @engine.connections[@conn] = self
+        @engine.emit_monitor_event(:handshake_succeeded, endpoint: @endpoint)
+        @engine.routing.connection_added(@conn)
+        @engine.peer_connected.resolve(@conn)
+        transition!(:ready)
+      end
+
+
+      def tear_down!(reconnect:)
+        return if @state == :closed
+        transition!(:closed)
+        @engine.connections.delete(@conn)
+        @engine.routing.connection_removed(@conn) if @conn
+        @conn&.close rescue nil
+        @engine.emit_monitor_event(:disconnected, endpoint: @endpoint)
+        @done&.resolve(true)
+        @engine.resolve_all_peers_gone_if_empty
+        @engine.maybe_reconnect(@endpoint) if reconnect
+      end
+
+
+      def transition!(new_state)
+        allowed = TRANSITIONS[@state]
+        unless allowed&.include?(new_state)
+          raise InvalidTransition, "#{@state} → #{new_state}"
+        end
+        @state = new_state
+      end
+    end
+  end
+end

data/lib/omq/engine/socket_lifecycle.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Owns the socket-level state: `:new → :open → :closing → :closed`,
+    # the first-peer / last-peer signaling promises, the reconnect flag,
+    # and the captured parent task for the socket's task tree.
+    #
+    # Engine delegates state queries here and uses it to coordinate the
+    # ordering of close-time side effects. This consolidates six ivars
+    # (`@state`, `@peer_connected`, `@all_peers_gone`, `@reconnect_enabled`,
+    # `@parent_task`, `@on_io_thread`) into one cohesive object with
+    # explicit transitions.
+    #
+    class SocketLifecycle
+      class InvalidTransition < RuntimeError; end
+
+      STATES = %i[new open closing closed].freeze
+
+      TRANSITIONS = {
+        new:     %i[open closed].freeze,
+        open:    %i[closing closed].freeze,
+        closing: %i[closed].freeze,
+        closed:  [].freeze,
+      }.freeze
+
+
+      # @return [Symbol]
+      attr_reader :state
+
+      # @return [Async::Promise] resolves with the first connected peer
+      attr_reader :peer_connected
+
+      # @return [Async::Promise] resolves once all peers are gone (after having had peers)
+      attr_reader :all_peers_gone
+
+      # @return [Async::Task, nil] root of the socket's task tree
+      attr_reader :parent_task
+
+      # @return [Boolean] true if parent_task is the shared Reactor thread
+      attr_reader :on_io_thread
+
+      # @return [Boolean] whether auto-reconnect is enabled
+      attr_accessor :reconnect_enabled
+
+
+      def initialize
+        @state             = :new
+        @peer_connected    = Async::Promise.new
+        @all_peers_gone    = Async::Promise.new
+        @reconnect_enabled = true
+        @parent_task       = nil
+        @on_io_thread      = false
+      end
+
+
+      def open?      = @state == :open
+      def closing?   = @state == :closing
+      def closed?    = @state == :closed
+      def alive?     = @state == :new || @state == :open
+
+
+      # Captures the current Async task (or the shared Reactor root) as
+      # this socket's task tree root. Transitions `:new → :open`.
+      #
+      # @param linger [Numeric, nil] used to register the Reactor linger slot
+      #   when falling back to the IO thread
+      # @return [Boolean] true on first-time capture, false if already captured
+      #
+      def capture_parent_task(linger:)
+        return false if @parent_task
+        if Async::Task.current?
+          @parent_task = Async::Task.current
+        else
+          @parent_task  = Reactor.root_task
+          @on_io_thread = true
+          Reactor.track_linger(linger)
+        end
+        transition!(:open)
+        true
+      end
+
+
+      # Transitions `:open → :closing`.
+      def start_closing!
+        transition!(:closing)
+      end
+
+
+      # Transitions `:closing → :closed` (or `:new → :closed` for
+      # never-opened sockets).
+      def finish_closing!
+        transition!(:closed)
+      end
+
+
+      # Resolves `all_peers_gone` if we had peers and now have none.
+      # @param connections [Hash] current connection map
+      def resolve_all_peers_gone_if_empty(connections)
+        return unless @peer_connected.resolved? && connections.empty?
+        @all_peers_gone.resolve(true)
+      end
+
+
+      private
+
+      def transition!(new_state)
+        allowed = TRANSITIONS[@state]
+        unless allowed&.include?(new_state)
+          raise InvalidTransition, "#{@state} → #{new_state}"
+        end
+        @state = new_state
+      end
+    end
+  end
+end

data/lib/omq/engine.rb

@@ -4,7 +4,8 @@ require "async"
 require_relative "engine/recv_pump"
 require_relative "engine/heartbeat"
 require_relative "engine/reconnect"
-require_relative "engine/connection_setup"
+require_relative "engine/connection_lifecycle"
+require_relative "engine/socket_lifecycle"
 require_relative "engine/maintenance"
 
 module OMQ
@@ -25,13 +26,6 @@ module OMQ
     end
 
 
-    # Per-connection metadata: the endpoint it was established on and an
-    # optional Promise resolved when the connection is lost (used by
-    # {#spawn_connection} to await connection teardown).
-    #
-    ConnectionRecord = Data.define(:endpoint, :done)
-
-
     # @return [Symbol] socket type (e.g. :REQ, :PAIR)
     #
     attr_reader :socket_type
@@ -63,46 +57,43 @@ module OMQ
     # @param options [Options]
     #
     def initialize(socket_type, options)
-      @socket_type       = socket_type
-      @options           = options
-      @routing           = nil
-      @connections       = {} # connection => ConnectionRecord
-      @dialed            = Set.new # endpoints we called connect() on (reconnect intent)
-      @listeners         = []
-      @tasks             = []
-      @state             = :open
-      @last_endpoint     = nil
-      @last_tcp_port     = nil
-      @peer_connected    = Async::Promise.new
-      @all_peers_gone    = Async::Promise.new
-      @reconnect_enabled = true
-      @parent_task       = nil
-      @on_io_thread      = false
-      @fatal_error       = nil
-      @monitor_queue     = nil
-      @verbose_monitor   = false
-    end
-
-
-    # @return [Async::Promise] resolves when first peer completes handshake
-    # @return [Async::Promise] resolves when all peers disconnect (after having had peers)
-    # @return [Hash{Connection => ConnectionRecord}] active connections
-    # @return [Async::Task, nil] root task for spawning subtrees
+      @socket_type     = socket_type
+      @options         = options
+      @routing         = nil
+      @connections     = {} # connection => ConnectionLifecycle
+      @dialed          = Set.new # endpoints we called connect() on (reconnect intent)
+      @listeners       = []
+      @tasks           = []
+      @lifecycle       = SocketLifecycle.new
+      @last_endpoint   = nil
+      @last_tcp_port   = nil
+      @fatal_error     = nil
+      @monitor_queue   = nil
+      @verbose_monitor = false
+    end
+
+
+    # @return [Hash{Connection => ConnectionLifecycle}] active connections
     # @return [Array<Async::Task>] background tasks (pumps, heartbeat, reconnect)
+    # @return [SocketLifecycle] socket-level state + signaling
     #
-    attr_reader :peer_connected, :all_peers_gone, :connections, :parent_task, :tasks
+    attr_reader :connections, :tasks, :lifecycle
 
-    # @!attribute [w] reconnect_enabled
-    #   @param value [Boolean] enable or disable auto-reconnect
     # @!attribute [w] monitor_queue
     #   @param value [Async::Queue, nil] queue for monitor events
     #
-    attr_writer :reconnect_enabled, :monitor_queue
+    attr_writer :monitor_queue
     attr_accessor :verbose_monitor
 
-    # @return [Boolean] true if the engine has been closed
-    #
-    def closed? = @state == :closed
+
+    # Delegated to {SocketLifecycle}.
+    def peer_connected    = @lifecycle.peer_connected
+    def all_peers_gone    = @lifecycle.all_peers_gone
+    def parent_task       = @lifecycle.parent_task
+    def closed?           = @lifecycle.closed?
+    def reconnect_enabled=(value)
+      @lifecycle.reconnect_enabled = value
+    end
 
     # Optional proc that wraps new connections (e.g. for serialization).
     # Called with the raw connection; must return the (possibly wrapped) connection.
@@ -110,7 +101,7 @@ module OMQ
     attr_accessor :connection_wrapper
 
 
-    # Spawns an inproc reconnect retry task under @parent_task.
+    # Spawns an inproc reconnect retry task under the socket's parent task.
     #
     # @param endpoint [String]
     # @yield [interval] the retry loop body
@@ -118,7 +109,7 @@ module OMQ
     def spawn_inproc_retry(endpoint)
       ri  = @options.reconnect_interval
       ivl = ri.is_a?(Range) ? ri.begin : ri
-      @tasks << @parent_task.async(transient: true, annotation: "inproc reconnect #{endpoint}") do
+      @tasks << @lifecycle.parent_task.async(transient: true, annotation: "inproc reconnect #{endpoint}") do
         yield ivl
       rescue Async::Stop
       end
@@ -224,11 +215,7 @@ module OMQ
     # @return [void]
     #
     def connection_ready(pipe, endpoint: nil)
-      pipe = @connection_wrapper.call(pipe) if @connection_wrapper
-      @connections[pipe] = ConnectionRecord.new(endpoint: endpoint, done: nil)
-      emit_monitor_event(:handshake_succeeded, endpoint: endpoint)
-      routing.connection_added(pipe)
-      @peer_connected.resolve(pipe)
+      ConnectionLifecycle.new(self, endpoint: endpoint).ready_direct!(pipe)
     end
 
 
@@ -309,13 +296,25 @@ module OMQ
     # @return [void]
     #
     def connection_lost(connection)
-      entry = @connections.delete(connection)
-      routing.connection_removed(connection)
-      connection.close
-      emit_monitor_event(:disconnected, endpoint: entry&.endpoint)
-      entry&.done&.resolve(true)
-      @all_peers_gone.resolve(true) if @peer_connected.resolved? && @connections.empty?
-      maybe_reconnect(entry&.endpoint)
+      @connections[connection]&.lost!
+    end
+
+
+    # Resolves `all_peers_gone` if we had peers and now have none.
+    # Called by ConnectionLifecycle during teardown.
+    #
+    def resolve_all_peers_gone_if_empty
+      @lifecycle.resolve_all_peers_gone_if_empty(@connections)
+    end
+
+
+    # Schedules a reconnect for +endpoint+ if auto-reconnect is enabled
+    # and the endpoint is still dialed.
+    #
+    def maybe_reconnect(endpoint)
+      return unless endpoint && @dialed.include?(endpoint)
+      return unless @lifecycle.open? && @lifecycle.reconnect_enabled
+      Reconnect.schedule(endpoint, @options, @lifecycle.parent_task, self)
     end
 
 
@@ -324,12 +323,12 @@ module OMQ
     # @return [void]
     #
     def close
-      return unless @state == :open
-      @state = :closing
+      return unless @lifecycle.open?
+      @lifecycle.start_closing!
       stop_listeners unless @connections.empty?
       drain_send_queues(@options.linger) if @options.linger.nil? || @options.linger > 0
-      @state = :closed
-      Reactor.untrack_linger(@options.linger) if @on_io_thread
+      @lifecycle.finish_closing!
+      Reactor.untrack_linger(@options.linger) if @lifecycle.on_io_thread
       stop_listeners
       close_connections
       stop_tasks
@@ -368,14 +367,14 @@ module OMQ
     # @param error [Exception]
     #
     def signal_fatal_error(error)
-      return unless @state == :open
+      return unless @lifecycle.open?
       @fatal_error = begin
         raise OMQ::SocketDeadError, "internal error killed #{@socket_type} socket"
       rescue => wrapped
         wrapped
       end
       routing.recv_queue.push(nil) rescue nil
-      @peer_connected.resolve(nil) rescue nil
+      @lifecycle.peer_connected.resolve(nil) rescue nil
     end
 
 
@@ -385,15 +384,8 @@ module OMQ
     # callers get the IO thread's root task, not an ephemeral work task.
     #
     def capture_parent_task
-      return if @parent_task
-      if Async::Task.current?
-        @parent_task = Async::Task.current
-      else
-        @parent_task  = Reactor.root_task
-        @on_io_thread = true
-        Reactor.track_linger(@options.linger)
-      end
-      Maintenance.start(@parent_task, @options.mechanism, @tasks)
+      return unless @lifecycle.capture_parent_task(linger: @options.linger)
+      Maintenance.start(@lifecycle.parent_task, @options.mechanism, @tasks)
     end
 
 
@@ -440,16 +432,17 @@ module OMQ
     private
 
     def spawn_connection(io, as_server:, endpoint: nil)
-      task = @parent_task&.async(transient: true, annotation: "conn #{endpoint}") do
-        done = Async::Promise.new
-        conn = ConnectionSetup.run(io, self, as_server: as_server, endpoint: endpoint, done: done)
+      task = @lifecycle.parent_task&.async(transient: true, annotation: "conn #{endpoint}") do
+        done      = Async::Promise.new
+        lifecycle = ConnectionLifecycle.new(self, endpoint: endpoint, done: done)
+        lifecycle.handshake!(io, as_server: as_server)
         done.wait
       rescue Async::Queue::ClosedError
         # connection dropped during drain — message re-staged
       rescue Protocol::ZMTP::Error, *CONNECTION_LOST
         # handshake failed or connection lost — subtree cleaned up
       ensure
-        conn&.close rescue nil
+        lifecycle&.close!
       end
       @tasks << task if task
     end
@@ -465,15 +458,8 @@ module OMQ
     end
 
 
-    def maybe_reconnect(endpoint)
-      return unless endpoint && @dialed.include?(endpoint)
-      return unless @state == :open && @reconnect_enabled
-      Reconnect.schedule(endpoint, @options, @parent_task, self)
-    end
-
-
     def schedule_reconnect(endpoint, delay: nil)
-      Reconnect.schedule(endpoint, @options, @parent_task, self, delay: delay)
+      Reconnect.schedule(endpoint, @options, @lifecycle.parent_task, self, delay: delay)
     end
 
 
@@ -485,7 +471,7 @@ module OMQ
 
     def start_accept_loops(listener)
       return unless listener.respond_to?(:start_accept_loops)
-      listener.start_accept_loops(@parent_task) do |io|
+      listener.start_accept_loops(@lifecycle.parent_task) do |io|
         handle_accepted(io, endpoint: listener.endpoint)
       end
     end
@@ -498,18 +484,12 @@ module OMQ
 
 
     def close_connections
-      @connections.each_key(&:close)
-      @connections.clear
+      @connections.values.each(&:close!)
     end
 
 
     def close_connections_at(endpoint)
-      conns = @connections.filter_map { |conn, e| conn if e.endpoint == endpoint }
-      conns.each do |conn|
-        @connections.delete(conn)
-        routing.connection_removed(conn)
-        conn.close
-      end
+      @connections.values.select { |lc| lc.endpoint == endpoint }.each(&:close!)
     end
 
 

data/lib/omq/routing/conn_send_pump.rb

@@ -21,7 +21,11 @@ module OMQ
           loop do
             batch = [q.dequeue]
             Routing.drain_send_queue(q, batch)
-            batch.each { |parts| conn.write_message(parts) }
+            if batch.size == 1
+              conn.write_message(batch[0])
+            else
+              conn.write_messages(batch)
+            end
             conn.flush
             batch.each { |parts| engine.emit_verbose_monitor_event(:message_sent, parts: parts) }
           rescue Protocol::ZMTP::Error, *CONNECTION_LOST

data/lib/omq/routing/fan_out.rb

@@ -83,7 +83,7 @@ module OMQ
       # @param conn [Connection]
       #
       def add_fan_out_send_connection(conn)
-        q = Routing.build_queue(@engine.options.send_hwm, :block)
+        q = Routing.build_queue(@engine.options.send_hwm, @engine.options.on_mute)
         @conn_queues[conn] = q
         start_conn_send_pump(conn, q)
       end
@@ -107,10 +107,11 @@ module OMQ
       # are respected: a message enqueued before the async subscription listener
       # has processed SUBSCRIBE commands will still be delivered correctly.
       #
-      # Per-connection queues use :block (Async::LimitedQueue) for
-      # backpressure: when a subscriber's queue is full, the publisher
-      # yields until the send pump drains it. This matches the old
-      # shared-queue behavior and keeps the publisher fiber-friendly.
+      # Per-connection queues honor the socket's on_mute strategy.
+      # PUB/XPUB/RADIO default to :drop_newest so one slow subscriber
+      # silently drops its own messages without stalling the publisher
+      # or other subscribers. Applications can opt in to :block for
+      # strict backpressure.
       #
       # @param parts [Array<String>]
       #

data/lib/omq/routing/pair.rb

@@ -4,9 +4,10 @@ module OMQ
   module Routing
     # PAIR socket routing: exclusive 1-to-1 bidirectional.
     #
-    # Only one peer connection is allowed. Send and recv queues are
-    # created per-connection (and destroyed on disconnection) so
-    # HWM is consistent with multi-peer socket types.
+    # Only one peer connection is allowed at a time. The send queue
+    # is socket-level (one shared bounded queue), and a single send
+    # pump fiber drains it into the connected peer. On disconnect,
+    # the in-flight batch is dropped (matching libzmq).
     #
     class Pair
       include FairRecv
@@ -14,13 +15,12 @@ module OMQ
       # @param engine [Engine]
       #
       def initialize(engine)
-        @engine         = engine
-        @connection     = nil
-        @recv_queue     = FairQueue.new
-        @send_queue     = nil   # created per-connection
-        @staging_queue  = StagingQueue.new(@engine.options.send_hwm)
-        @send_pump      = nil
-        @tasks          = []
+        @engine     = engine
+        @connection = nil
+        @recv_queue = FairQueue.new
+        @send_queue = Routing.build_queue(@engine.options.send_hwm, :block)
+        @send_pump  = nil
+        @tasks      = []
       end
 
 
@@ -38,10 +38,6 @@ module OMQ
         add_fair_recv_connection(connection)
 
         unless connection.is_a?(Transport::Inproc::DirectPipe)
-          @send_queue = Routing.build_queue(@engine.options.send_hwm, :block)
-          while (msg = @staging_queue.dequeue)
-            @send_queue.enqueue(msg)
-          end
           start_send_pump(connection)
         end
       end
@@ -53,12 +49,6 @@ module OMQ
         if @connection == connection
           @connection = nil
           @recv_queue.remove_queue(connection)
-          if @send_queue
-            while (msg = @send_queue.dequeue(timeout: 0))
-              @staging_queue.prepend(msg)
-            end
-            @send_queue = nil
-          end
           @send_pump&.stop
           @send_pump = nil
         end
@@ -71,10 +61,8 @@ module OMQ
         conn = @connection
         if conn.is_a?(Transport::Inproc::DirectPipe) && conn.direct_recv_queue
           conn.send_message(parts)
-        elsif @send_queue
-          @send_queue.enqueue(parts)
         else
-          @staging_queue.enqueue(parts)
+          @send_queue.enqueue(parts)
         end
       end
 
@@ -89,10 +77,10 @@ module OMQ
       end
 
 
-      # @return [Boolean] true when the staging and send queues are empty
+      # @return [Boolean] true when the shared send queue is empty
       #
       def send_queues_drained?
-        @staging_queue.empty? && (@send_queue.nil? || @send_queue.empty?)
+        @send_queue.empty?
       end
 
       private
@@ -102,15 +90,16 @@ module OMQ
           loop do
             batch = [@send_queue.dequeue]
             Routing.drain_send_queue(@send_queue, batch)
-            begin
-              batch.each { |parts| conn.write_message(parts) }
-              conn.flush
-              batch.each { |parts| @engine.emit_verbose_monitor_event(:message_sent, parts: parts) }
-            rescue Protocol::ZMTP::Error, *CONNECTION_LOST
-              batch.each { |parts| @staging_queue.prepend(parts) }
-              @engine.connection_lost(conn)
-              break
+            if batch.size == 1
+              conn.write_message(batch[0])
+            else
+              conn.write_messages(batch)
             end
+            conn.flush
+            batch.each { |parts| @engine.emit_verbose_monitor_event(:message_sent, parts: parts) }
+          rescue Protocol::ZMTP::Error, *CONNECTION_LOST
+            @engine.connection_lost(conn)
+            break
           end
         end
         @tasks << @send_pump

data/lib/omq/routing/round_robin.rb

@@ -2,95 +2,69 @@
 
 module OMQ
   module Routing
-    # Mixin for routing strategies that send via round-robin.
+    # Mixin for routing strategies that load-balance via work-stealing.
     #
-    # Provides reactive connection management: Async::Promise waits
-    # for the first connection, Array#cycle handles round-robin,
-    # and a new Promise is created when all connections drop.
+    # One shared bounded send queue per socket (`send_hwm` enforced
+    # at the socket level, not per peer). Each connected peer gets its
+    # own send pump fiber that races to drain the shared queue. Slow
+    # peers' pumps naturally block on their own TCP flush; fast peers'
+    # pumps keep dequeuing. The result is work-stealing load balancing,
+    # which is strictly better than libzmq's strict per-pipe round-robin
+    # for PUSH-style patterns.
     #
-    # Each connected peer gets its own bounded send queue and a
-    # dedicated send pump fiber, ensuring HWM is enforced per peer.
+    # See DESIGN.md "Per-socket HWM (not per-connection)" for the
+    # full reasoning.
     #
     # Including classes must call `init_round_robin(engine)` from
     # their #initialize.
     #
     module RoundRobin
-      # @return [Boolean] true when the staging queue and all per-connection
-      #   send queues are empty
+      # @return [Boolean] true when the shared send queue is empty
       #
       def send_queues_drained?
-        @staging_queue.empty? && @conn_queues.values.all?(&:empty?)
+        @send_queue.empty?
       end
 
       private
 
-      # Initializes round-robin state for the including class.
+      # Initializes the shared send queue for the including class.
       #
       # @param engine [Engine]
       #
       def init_round_robin(engine)
-        @connections          = []
-        @cycle                = @connections.cycle
-        @connection_available = Async::Promise.new
-        @conn_queues          = {}  # connection => send queue
-        @conn_send_tasks      = {}  # connection => send pump task
-        @direct_pipe          = nil
-        @staging_queue        = StagingQueue.new(@engine.options.send_hwm)
+        @connections     = []
+        @send_queue      = Routing.build_queue(engine.options.send_hwm, :block)
+        @direct_pipe     = nil
+        @conn_send_tasks = {}  # conn => send pump task
       end
 
 
-      # Creates a per-connection send queue and starts its send pump.
-      # Call from #connection_added after appending to @connections.
+      # Registers a connection and starts its send pump.
+      # Call from #connection_added.
       #
       # @param conn [Connection]
       #
       def add_round_robin_send_connection(conn)
-        q = Routing.build_queue(@engine.options.send_hwm, :block)
-        @conn_queues[conn] = q
-        start_conn_send_pump(conn, q)
-        drain_staging_to(q)
         @connections << conn
         update_direct_pipe
-        # A message may have squeezed into staging while drain was
-        # running (the pipe loop's blocked enqueue completed after
-        # drain freed space).  Now that @connections is set, no new
-        # messages will go to staging, so this second drain is
-        # guaranteed to finish quickly.
-        drain_staging_to(q)
-        signal_connection_available
+        start_conn_send_pump(conn)
       end
 
 
-      # Stops the per-connection send pump and removes the queue.
-      # Call from #connection_removed.
+      # Removes the connection and stops its send pump. Any message
+      # the pump had already dequeued but not yet written is dropped --
+      # matching libzmq's behavior on `pipe_terminated`. PUSH has no
+      # cross-peer ordering guarantee, so this is safe.
       #
       # @param conn [Connection]
       #
       def remove_round_robin_send_connection(conn)
         update_direct_pipe
-        q = @conn_queues.delete(conn)
-        if q
-          while (msg = q.dequeue(timeout: 0))
-            @staging_queue.prepend(msg)
-          end
-          q.close
-        end
-        @conn_send_tasks.delete(conn)
-      end
-
-
-      # Resolves the connection-available promise so blocked
-      # senders can proceed.
-      #
-      def signal_connection_available
-        unless @connection_available.resolved?
-          @connection_available.resolve(true)
-        end
+        @conn_send_tasks.delete(conn)&.stop
       end
 
 
       # Updates the direct-pipe shortcut for inproc single-peer bypass.
-      # Call from connection_added after @connections is updated.
       #
       def update_direct_pipe
         if @connections.size == 1 && @connections.first.is_a?(Transport::Inproc::DirectPipe)
@@ -101,57 +75,17 @@ module OMQ
       end
 
 
-      # Enqueues directly to the inproc peer's recv queue if possible.
-      # When peers are connected, picks the next one round-robin and
-      # enqueues into its per-connection send queue (blocking if full).
-      # When no peers are connected yet, buffers in a staging queue
-      # (bounded by send_hwm) — drained into the first peer's queue
-      # when it connects.
+      # Enqueues a message. For inproc single-peer, bypasses the queue
+      # and writes directly to the peer's recv queue. Otherwise blocks
+      # on the shared bounded send queue (backpressure when full).
       #
       def enqueue_round_robin(parts)
         pipe = @direct_pipe
         if pipe&.direct_recv_queue
           pipe.send_message(transform_send(parts))
-        elsif @connections.empty?
-          @staging_queue.enqueue(parts)
         else
-          conn = next_connection
-          @conn_queues[conn].enqueue(parts)
-        end
-      rescue Async::Queue::ClosedError
-        retry
-      end
-
-
-      # Drains the staging queue into the given per-connection queue.
-      # Called when the first peer connects, to deliver messages that
-      # were enqueued before any connection existed.
-      #
-      def drain_staging_to(q)
-        while (msg = @staging_queue.dequeue)
-          q.enqueue(msg)
+          @send_queue.enqueue(parts)
         end
-      rescue Async::Queue::ClosedError
-        # Connection dropped while draining — put the undelivered
-        # message back at the front so ordering is preserved.
-        @staging_queue.prepend(msg) if msg
-      end
-
-
-      # Blocks until a connection is available, then returns
-      # the next one in round-robin order.
-      #
-      # @return [Connection]
-      #
-      def next_connection
-        @cycle.next
-      rescue StopIteration
-        if @connections.empty?
-          @connection_available = Async::Promise.new
-          @connection_available.wait
-        end
-        @cycle = @connections.cycle
-        retry
       end
 
 
@@ -164,24 +98,30 @@ module OMQ
       def transform_send(parts) = parts
 
 
-      # Starts a dedicated send pump for one connection.
-      # Batches messages for throughput; flushes after each batch.
-      # Calls Engine#connection_lost on disconnect so reconnect fires.
+      # Spawns a send pump for one connection. Drains the shared send
+      # queue, writes to its peer, and yields. On disconnect, the
+      # in-flight batch is dropped and the engine reconnect kicks in.
+      #
+      # Each batch is capped at BATCH_MSG_CAP messages OR BATCH_BYTE_CAP
+      # bytes, whichever hits first. The cap exists for fairness when
+      # multiple peers share the queue: without it, the first pump that
+      # wakes up drains the entire queue in one non-blocking burst
+      # before any other pump runs (TCP send buffers absorb bursts
+      # without forcing a fiber yield). 512 KB lets large-message
+      # workloads batch naturally (8 × 64 KB per batch) while keeping
+      # per-pump latency bounded enough that small-message multi-peer
+      # fairness still benefits.
       #
       # @param conn [Connection]
-      # @param q [Async::LimitedQueue] the connection's send queue
       #
-      def start_conn_send_pump(conn, q)
+      def start_conn_send_pump(conn)
         task = @engine.spawn_pump_task(annotation: "send pump") do
           loop do
-            msg = q.dequeue
-            break unless msg  # queue closed by remove_round_robin_send_connection
-            batch = [msg]
-            Routing.drain_send_queue(q, batch)
+            batch = [@send_queue.dequeue]
+            drain_send_queue_capped(batch)
             write_batch(conn, batch)
             batch.each { |parts| @engine.emit_verbose_monitor_event(:message_sent, parts: parts) }
           rescue Protocol::ZMTP::Error, *CONNECTION_LOST
-            batch&.each { |parts| @staging_queue.prepend(parts) }
             @engine.connection_lost(conn)
             break
           end
@@ -191,11 +131,28 @@ module OMQ
       end
 
 
+      BATCH_MSG_CAP  = 256
+      BATCH_BYTE_CAP = 512 * 1024
+
+      def drain_send_queue_capped(batch)
+        bytes = batch[0].sum(&:bytesize)
+        while batch.size < BATCH_MSG_CAP && bytes < BATCH_BYTE_CAP
+          msg = @send_queue.dequeue(timeout: 0)
+          break unless msg
+          batch << msg
+          bytes += msg.sum(&:bytesize)
+        end
+      end
+
+
       def write_batch(conn, batch)
         if batch.size == 1
           conn.send_message(transform_send(batch[0]))
         else
-          batch.each { |parts| conn.write_message(transform_send(parts)) }
+          # Single mutex acquisition for the whole batch (up to
+          # BATCH_MSG_CAP messages). transform_send is identity for
+          # most routings and only allocates a new parts array for REQ.
+          conn.write_messages(batch.map { |parts| transform_send(parts) })
           conn.flush
         end
       end

data/lib/omq/routing.rb

@@ -4,7 +4,6 @@ require "async"
 require "async/queue"
 require "async/limited_queue"
 require_relative "drop_queue"
-require_relative "routing/staging_queue"
 require_relative "routing/fair_queue"
 require_relative "routing/fair_recv"
 require_relative "routing/conn_send_pump"

data/lib/omq/transport/inproc/direct_pipe.rb

@@ -96,6 +96,18 @@ module OMQ
         alias write_message send_message
 
 
+        # Batched form, for parity with Protocol::ZMTP::Connection. The
+        # work-stealing pumps call this when they dequeue more than one
+        # message at once; DirectPipe just loops — no mutex to amortize.
+        #
+        # @param messages [Array<Array<String>>]
+        # @return [void]
+        #
+        def write_messages(messages)
+          messages.each { |parts| send_message(parts) }
+        end
+
+
         # @return [Boolean] always false; inproc pipes are never encrypted
         #
         def encrypted? = false

data/lib/omq/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module OMQ
-  VERSION = "0.15.5"
+  VERSION = "0.16.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq
 version: !ruby/object:Gem::Version
-  version: 0.15.5
+  version: 0.16.1
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '0.4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.3'
+        version: '0.4'
 - !ruby/object:Gem::Dependency
   name: async
   requirement: !ruby/object:Gem::Requirement
@@ -67,11 +67,12 @@ files:
 - lib/omq.rb
 - lib/omq/drop_queue.rb
 - lib/omq/engine.rb
-- lib/omq/engine/connection_setup.rb
+- lib/omq/engine/connection_lifecycle.rb
 - lib/omq/engine/heartbeat.rb
 - lib/omq/engine/maintenance.rb
 - lib/omq/engine/reconnect.rb
 - lib/omq/engine/recv_pump.rb
+- lib/omq/engine/socket_lifecycle.rb
 - lib/omq/monitor_event.rb
 - lib/omq/options.rb
 - lib/omq/pair.rb
@@ -96,7 +97,6 @@ files:
 - lib/omq/routing/req.rb
 - lib/omq/routing/round_robin.rb
 - lib/omq/routing/router.rb
-- lib/omq/routing/staging_queue.rb
 - lib/omq/routing/sub.rb
 - lib/omq/routing/xpub.rb
 - lib/omq/routing/xsub.rb

data/lib/omq/engine/connection_setup.rb

@@ -1,70 +0,0 @@
-# frozen_string_literal: true
-
-module OMQ
-  class Engine
-    # Performs ZMTP handshake and registers a new connection.
-    #
-    class ConnectionSetup
-      # @param io [#read, #write, #close] underlying transport stream
-      # @param engine [Engine]
-      # @param as_server [Boolean]
-      # @param endpoint [String, nil]
-      # @param done [Async::Promise, nil] resolved when connection is lost
-      # @return [Connection]
-      #
-      def self.run(io, engine, as_server:, endpoint: nil, done: nil)
-        new(engine).run(io, as_server: as_server, endpoint: endpoint, done: done)
-      end
-
-
-      # @param engine [Engine]
-      #
-      def initialize(engine)
-        @engine = engine
-      end
-
-
-      # Performs the ZMTP handshake, starts heartbeat, and registers the connection.
-      #
-      # @param io [#read, #write, #close]
-      # @param as_server [Boolean]
-      # @param endpoint [String, nil]
-      # @param done [Async::Promise, nil] resolved when connection is lost
-      # @return [Connection]
-      #
-      def run(io, as_server:, endpoint: nil, done: nil)
-        conn = build_connection(io, as_server)
-        conn.handshake!
-        Heartbeat.start(Async::Task.current, conn, @engine.options, @engine.tasks)
-        conn = @engine.connection_wrapper.call(conn) if @engine.connection_wrapper
-        @engine.emit_monitor_event(:handshake_succeeded, endpoint: endpoint)
-        register(conn, endpoint, done)
-        conn
-      rescue Protocol::ZMTP::Error, *CONNECTION_LOST => error
-        @engine.emit_monitor_event(:handshake_failed, endpoint: endpoint, detail: { error: error })
-        conn&.close
-        raise
-      end
-
-      private
-
-      def build_connection(io, as_server)
-        Protocol::ZMTP::Connection.new(
-          io,
-          socket_type:      @engine.socket_type.to_s,
-          identity:         @engine.options.identity,
-          as_server:        as_server,
-          mechanism:        @engine.options.mechanism&.dup,
-          max_message_size: @engine.options.max_message_size,
-        )
-      end
-
-
-      def register(conn, endpoint, done)
-        @engine.connections[conn] = Engine::ConnectionRecord.new(endpoint: endpoint, done: done)
-        @engine.routing.connection_added(conn)
-        @engine.peer_connected.resolve(conn)
-      end
-    end
-  end
-end

data/lib/omq/routing/staging_queue.rb

@@ -1,66 +0,0 @@
-# frozen_string_literal: true
-
-module OMQ
-  module Routing
-    # Bounded FIFO queue for staging unsent messages.
-    #
-    # Wraps an +Async::LimitedQueue+ for backpressure, with a small
-    # prepend buffer checked first on dequeue (same trick as the
-    # prefetch buffer in {OMQ::Readable#receive}).
-    #
-    class StagingQueue
-      # @param max [Integer, nil] capacity (nil or 0 = unbounded)
-      #
-      def initialize(max = nil)
-        @max   = (max && max > 0) ? max : nil
-        @queue = @max ? Async::LimitedQueue.new(@max) : Async::Queue.new
-        @head  = []
-        @mu    = Mutex.new
-      end
-
-
-      # Appends a message to the back.
-      # Blocks (fiber-yields) when at capacity.
-      #
-      # @param msg [Array<String>]
-      # @return [void]
-      #
-      def enqueue(msg)
-        @queue.enqueue(msg)
-      end
-
-
-      # Inserts a message at the front (for re-staging after a
-      # failed drain).  Drops the message if the staging queue is
-      # already at capacity (messages sent to a peer that disconnected
-      # may be lost -- same as ZMQ).
-      #
-      # @param msg [Array<String>]
-      # @return [void]
-      #
-      def prepend(msg)
-        @mu.synchronize do
-          return if @max && @head.size >= @max
-          @head.push(msg)
-        end
-      end
-
-
-      # Returns the first message: from the prepend buffer if
-      # non-empty, otherwise non-blocking dequeue from the main queue.
-      #
-      # @return [Array<String>, nil]
-      #
-      def dequeue
-        @mu.synchronize { @head.shift } || @queue.dequeue(timeout: 0)
-      end
-
-
-      # @return [Boolean]
-      #
-      def empty?
-        @mu.synchronize { @head.empty? } && @queue.empty?
-      end
-    end
-  end
-end