omq 0.11.0 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b736b6b715f5c4f330d21be278450600b2642239b8bea9e85957f516e89211ed
-  data.tar.gz: 34c057a32055e7333e4299e85c0fee9e798b43ea34d021f237ba6778338b65f1
+  metadata.gz: 91e6db2b4fd881530030f63c0c47a534c3f6361497d0f55ad1d28c7dbb85a669
+  data.tar.gz: 1af1d5692333586b650f7d0f14118ae04f64a65ab3d127d8ea3a3781e0eaae34
 SHA512:
-  metadata.gz: e95a6df6d4eb56ac1fc0586f61cfdb77d0afe9de620d3e540ffbae1256c2fe6526556485215eda3de20c8f34a332f51a68e5bb075087665049c0219bb07f6960
-  data.tar.gz: 8b5c8d38b7523e9c6ab30e17c21b0cf4adb2086e1bdde90298f62e42f4b8f2336cdd95c47293c3fb24fa41f376aed8a2ec6d1121fc56d41aa83d4afdc289d707
+  metadata.gz: 4a724ddc29600b8d101046ca82869e4598f733564be4022fc8eb287084a80319a1586d5e0ced449f3eea3a26127add8f962fc3914177cc95f4f495e98f87e7d4
+  data.tar.gz: '08935cd0ad4548af6a526984d84514b242ab1d29bfd14fb8933f7b84b9f906f1197588f6bc0fec95332d2dbac6e016308cbdb5bf5efe9e2cba7b2bbf7d695d58'

data/CHANGELOG.md

@@ -1,5 +1,148 @@
 # Changelog
 
+## 0.13.0
+
+### Changed
+
+- **`Engine` internals: `ConnectionRecord` + lifecycle state** — three parallel
+  per-connection ivars (`@connections` Array, `@connection_endpoints`,
+  `@connection_promises`) replaced by a single `@connections` Hash keyed by
+  connection, with values `ConnectionRecord = Data.define(:endpoint, :done)`.
+  `@connected_endpoints` renamed to `@dialed` (`Set`). `@closed`/`@closing`
+  booleans replaced by a `@state` symbol (`:open`/`:closing`/`:closed`).
+  Net: −4 instance variables.
+- **`@connections` in `FanOut`, `Sub`, `XSub` routing strategies changed from
+  `Array` to `Set`** — O(1) `#delete` on peer disconnect; semantics already
+  required uniqueness.
+
+### Fixed
+
+- **FanOut send queues no longer drop messages** — per-connection send queues in
+  `FanOut` (PUB/XPUB/RADIO) used `DropQueue` (`Thread::SizedQueue`) which never
+  blocked the publisher fiber. When burst-sending beyond `send_hwm`, the sender
+  ran without yielding and messages were silently dropped. Switched to
+  `Async::LimitedQueue` (`:block`) so the publisher yields when a per-connection
+  queue is full, giving the send pump fiber a chance to drain it.
+
+### Changed
+
+- **Benchmark suite redesign** — replaced ASCII plots (unicode_plot) with JSONL
+  result storage and a colored terminal regression report. Results are appended
+  to `bench/results.jsonl` (gitignored, machine-local). New commands:
+  `ruby bench/run_all.rb` (run all patterns), `ruby bench/report.rb` (compare
+  last runs, highlight regressions/improvements).
+
+### Added
+
+- **Per-peer HWM** — send and receive high-water marks now apply per connected
+  peer (RFC 28/29/30). Each peer gets its own bounded send queue and its own
+  bounded recv queue. A slow or muted peer no longer steals capacity from
+  other peers. `FairQueue` + `SignalingQueue` aggregate per-connection recv
+  queues with fair round-robin delivery; `RoundRobin` and `FanOut` mixins
+  maintain per-connection send queues with dedicated send pump fibers.
+  `PUSH`/`DEALER`/`PAIR` buffer messages in a staging queue when no peers are
+  connected yet, draining into the first peer's queue on connect.
+- **`FairQueue`** — new aggregator class (`lib/omq/routing/fair_queue.rb`)
+  that fair-queues across per-connection bounded queues. Pending messages from
+  a disconnected peer are drained before the queue is discarded.
+- **`Socket.bind` / `Socket.connect` class-method fix** — now pass the
+  endpoint via `@`/`>` prefix into the constructor so any post-attach
+  initialization in subclasses (e.g. XSUB's `subscribe:` kwarg) runs after
+  the connection is established.
+
+
+
+- **QoS infrastructure** — `Options#qos` attribute (default 0) and inproc
+  command queue support for QoS-enabled connections. The
+  [omq-qos](https://github.com/paddor/omq-qos) gem activates delivery
+  guarantees via prepends.
+- **REQ send/recv ordering** — REQ sockets now enforce strict
+  send/recv/send/recv alternation. Calling `#send` twice without a
+  `#receive` in between raises `SocketError`.
+- **DirectPipe command frame support** — `DirectPipe#receive_message`
+  accepts a block for command frames, matching the `Protocol::ZMTP::Connection`
+  interface. Enables inproc transports to handle ACK/NACK and other
+  command-level protocols.
+
+### Fixed
+
+- **`send_pump_idle?` visibility** — moved above `private` in `RoundRobin`
+  and `FanOut` so `Engine#drain_send_queues` can call it during socket close.
+
+- **`Socket#monitor`** — observe connection lifecycle events via a
+  block-based API. Returns an `Async::Task` that yields `MonitorEvent`
+  (Data.define) instances for `:listening`, `:accepted`, `:connected`,
+  `:connect_delayed`, `:connect_retried`, `:handshake_succeeded`,
+  `:handshake_failed`, `:accept_failed`, `:bind_failed`, `:disconnected`,
+  `:closed`, and `:monitor_stopped`. Event types align with libzmq's
+  `zmq_socket_monitor` where applicable. Pattern-matchable, zero overhead
+  when no monitor is attached.
+- **Pluggable transport registry** — `Engine.transports` is a scheme →
+  module hash. Built-in transports (`tcp`, `ipc`, `inproc`) are registered
+  at load time. External gems register via
+  `OMQ::Engine.transports["scheme"] = MyTransport`. Each transport
+  implements `.bind(endpoint, engine)` → Listener, `.connect(endpoint,
+  engine)`, and optionally `.validate_endpoint!(endpoint)`. Listeners
+  implement `#start_accept_loops(parent_task, &on_accepted)`, `#stop`,
+  `#endpoint`, and optionally `#port`.
+- **Mutable error lists** — `CONNECTION_LOST` and `CONNECTION_FAILED` are
+  no longer frozen at load time. Transport plugins can append error classes
+  (e.g. `OpenSSL::SSL::SSLError`) before the first `#bind`/`#connect`,
+  which freezes both arrays.
+
+- **`on_mute` option** — controls behavior when a socket enters the mute state
+  (HWM full). PUB, XPUB, and RADIO default to `on_mute: :drop_newest` — slow
+  subscribers are skipped in the fan-out rather than blocking the publisher.
+  SUB, XSUB, and DISH accept `on_mute: :drop_newest` or `:drop_oldest` to
+  drop messages on the receive side instead of applying backpressure. All other
+  socket types default to `:block` (existing behavior).
+- **`DropQueue`** — bounded queue with `:drop_newest` (tail drop) and
+  `:drop_oldest` (head drop) strategies. Used by recv queues when `on_mute`
+  is a drop strategy.
+- **`Routing.build_queue`** — factory method for building send/recv queues
+  based on HWM and mute strategy. Supports HWM of `0` or `nil` for unbounded
+  queues.
+
+### Changed
+
+- **`max_message_size` defaults to 1 MiB** — frames exceeding this limit cause
+  the connection to be dropped before the body is read from the wire, preventing
+  a malicious peer from causing arbitrary memory allocation. Set `socket.max_message_size = nil`
+  to restore the previous unlimited behavior.
+- **Accept loops moved into Listeners** — `TCP::Listener` and
+  `IPC::Listener` now own their accept loop logic via
+  `#start_accept_loops(parent_task, &on_accepted)`. Engine delegates
+  via duck-type check. This enables external transports to define
+  custom accept behavior without modifying Engine.
+- `Engine#transport_for` uses registry lookup instead of `case/when`.
+- `Engine#validate_endpoint!` delegates to transport module.
+- `Engine#bind` reads `listener.port` instead of parsing the endpoint
+  string.
+
+### Removed
+
+- **Draft socket types extracted** — `RADIO`, `DISH`, `CLIENT`, `SERVER`,
+  `SCATTER`, `GATHER`, `CHANNEL`, and `PEER` are no longer bundled with `omq`.
+  Use the [omq-draft](https://github.com/paddor/omq-draft) gem and require
+  the relevant entry point (`omq/draft/radiodish`, `omq/draft/clientserver`,
+  etc.).
+- **UDP transport extracted** — `udp://` endpoints are provided by
+  `omq-draft` (via `require "omq/draft/radiodish"`). No longer registered by
+  default.
+- **`Routing.for` plugin registry** — draft socket type removal added
+  `Routing.register(socket_type, strategy_class)` for external gems to
+  register routing strategies. Unknown types fall through the built-in
+  `case` to this registry before raising `ArgumentError`.
+
+- **TLS transport** — extracted to the
+  [omq-transport-tls](https://github.com/paddor/omq-transport-tls) gem.
+  (Experimental) `require "omq/transport/tls"` to restore `tls+tcp://` support.
+- `tls_context` / `tls_context=` removed from `Options` and `Socket`
+  (provided by omq-transport-tls).
+- `OpenSSL::SSL::SSLError` removed from `CONNECTION_LOST` (added back
+  by omq-transport-tls).
+- TLS benchmark transport removed from `bench_helper.rb` and `plot.rb`.
+
 ## 0.11.0
 
 ### Added

data/README.md

@@ -147,7 +147,7 @@ end
 
 ## Socket Types
 
-All sockets are thread-safe. Default HWM is 1000 messages per socket. Classes live under `OMQ::` (alias: `ØMQ`).
+All sockets are thread-safe. Default HWM is 1000 messages per socket. Default `max_message_size` is **1 MiB** — frames larger than this cause the connection to be dropped before the body is read from the wire. Set `socket.max_message_size = nil` to disable the limit or raise it as needed. Classes live under `OMQ::` (alias: `ØMQ`).
 
 #### Standard (multipart messages)
 
@@ -162,6 +162,8 @@ All sockets are thread-safe. Default HWM is 1000 messages per socket. Classes li
 
 #### Draft (single-frame only)
 
+These require the `omq-draft` gem.
+
 | Pattern | Send | Receive | When HWM full |
 |---------|------|---------|---------------|
 | **CLIENT** / **SERVER** | Round-robin / routing-ID | Fair-queue | Block |

data/lib/omq/drop_queue.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module OMQ
+  # A bounded queue that drops messages when full instead of blocking.
+  #
+  # Two drop strategies:
+  #   :drop_newest — discard the incoming message (tail drop)
+  #   :drop_oldest — discard the head, then enqueue the new message
+  #
+  # Used by SUB/XSUB/DISH recv queues when on_mute is a drop strategy.
+  #
+  class DropQueue
+    # @param limit [Integer] maximum number of items
+    # @param strategy [Symbol] :drop_newest or :drop_oldest
+    #
+    def initialize(limit, strategy: :drop_newest)
+      @queue    = Thread::SizedQueue.new(limit)
+      @strategy = strategy
+    end
+
+    # Enqueues an item. Drops according to the configured strategy if full.
+    #
+    # @param item [Object]
+    # @return [void]
+    #
+    def enqueue(item)
+      @queue.push(item, true)
+    rescue ThreadError
+      return if @strategy == :drop_newest
+
+      # :drop_oldest — discard head, enqueue new
+      @queue.pop(true) rescue nil
+      retry
+    end
+
+    # Removes and returns the next item, blocking if empty.
+    #
+    # @return [Object]
+    #
+    def dequeue(timeout: nil)
+      if timeout
+        @queue.pop(timeout: timeout)
+      else
+        @queue.pop
+      end
+    end
+
+    # @return [Boolean]
+    #
+    def empty?
+      @queue.empty?
+    end
+  end
+end

data/lib/omq/engine/connection_setup.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Performs ZMTP handshake and registers a new connection.
+    #
+    class ConnectionSetup
+      # @param io [#read, #write, #close] underlying transport stream
+      # @param engine [Engine]
+      # @param as_server [Boolean]
+      # @param endpoint [String, nil]
+      # @param done [Async::Promise, nil] resolved when connection is lost
+      # @return [Connection]
+      #
+      def self.run(io, engine, as_server:, endpoint: nil, done: nil)
+        conn = build_connection(io, engine, as_server)
+        conn.handshake!
+        Heartbeat.start(engine.parent_task, conn, engine.options, engine.tasks)
+        conn = engine.connection_wrapper.call(conn) if engine.connection_wrapper
+        register(conn, engine, endpoint, done)
+        engine.emit_monitor_event(:handshake_succeeded, endpoint: endpoint)
+        conn
+      rescue Protocol::ZMTP::Error, *CONNECTION_LOST => error
+        engine.emit_monitor_event(:handshake_failed, endpoint: endpoint, detail: { error: error })
+        conn&.close
+        raise
+      end
+
+      def self.build_connection(io, engine, as_server)
+        Protocol::ZMTP::Connection.new(
+          io,
+          socket_type:      engine.socket_type.to_s,
+          identity:         engine.options.identity,
+          as_server:        as_server,
+          mechanism:        engine.options.mechanism&.dup,
+          max_message_size: engine.options.max_message_size,
+        )
+      end
+
+      def self.register(conn, engine, endpoint, done)
+        engine.connections[conn] = Engine::ConnectionRecord.new(endpoint: endpoint, done: done)
+        engine.routing.connection_added(conn)
+        engine.peer_connected.resolve(conn)
+      end
+    end
+  end
+end

data/lib/omq/engine/heartbeat.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Spawns a heartbeat task for a connection.
+    #
+    # Sends PING frames at +interval+ seconds and closes the connection
+    # if no traffic is seen within +timeout+ seconds.
+    #
+    class Heartbeat
+      # @param parent_task [Async::Task]
+      # @param conn [Connection]
+      # @param options [Options]
+      # @param tasks [Array]
+      #
+      def self.start(parent_task, conn, options, tasks)
+        interval = options.heartbeat_interval
+        return unless interval
+
+        ttl     = options.heartbeat_ttl     || interval
+        timeout = options.heartbeat_timeout || interval
+        conn.touch_heartbeat
+
+        tasks << parent_task.async(transient: true, annotation: "heartbeat") do
+          loop do
+            sleep interval
+            conn.send_command(Protocol::ZMTP::Codec::Command.ping(ttl: ttl, context: "".b))
+            if conn.heartbeat_expired?(timeout)
+              conn.close
+              break
+            end
+          end
+        rescue Async::Stop
+        rescue *CONNECTION_LOST
+          # connection closed
+        end
+      end
+    end
+  end
+end

data/lib/omq/engine/reconnect.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Schedules reconnect attempts with exponential back-off.
+    #
+    # Runs a background task that loops until a connection is established
+    # or the engine is closed.
+    #
+    class Reconnect
+      # @param endpoint [String]
+      # @param options [Options]
+      # @param parent_task [Async::Task]
+      # @param engine [Engine] for transport_for / emit_monitor_event / signal_fatal_error / closed?
+      # @param delay [Numeric, nil] initial delay (defaults to reconnect_interval)
+      #
+      def self.schedule(endpoint, options, parent_task, engine, delay: nil)
+        ri = options.reconnect_interval
+        delay, max_delay = init_delay(ri, delay)
+
+        engine.tasks << parent_task.async(transient: true, annotation: "reconnect #{endpoint}") do
+          loop do
+            break if engine.closed?
+            sleep delay if delay > 0
+            break if engine.closed?
+            begin
+              engine.transport_for(endpoint).connect(endpoint, engine)
+              break
+            rescue *CONNECTION_LOST, *CONNECTION_FAILED, Protocol::ZMTP::Error
+              delay = next_delay(delay, max_delay, ri)
+              engine.emit_monitor_event(:connect_retried, endpoint: endpoint, detail: { interval: delay })
+            end
+          end
+        rescue Async::Stop
+        rescue => error
+          engine.signal_fatal_error(error)
+        end
+      end
+
+      def self.init_delay(ri, delay)
+        if ri.is_a?(Range)
+          [delay || ri.begin, ri.end]
+        else
+          [delay || ri, nil]
+        end
+      end
+
+      def self.next_delay(delay, max_delay, ri)
+        delay = delay * 2
+        delay = [delay, max_delay].min if max_delay
+        delay = (ri.is_a?(Range) ? ri.begin : ri) if delay == 0
+        delay
+      end
+    end
+  end
+end

data/lib/omq/engine/recv_pump.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Starts a recv pump for a connection.
+    #
+    # For inproc DirectPipe: wires the direct recv path (no fiber spawned).
+    # For TCP/IPC: spawns a transient task that reads messages from the
+    # connection and enqueues them into +recv_queue+.
+    #
+    # The two-branch structure (with/without transform) is intentional for
+    # YJIT: it gives the JIT a monomorphic call per routing strategy instead
+    # of a megamorphic `transform.call` dispatch inside a shared loop.
+    #
+    # @param parent_task [Async::Task]
+    # @param conn [Connection, Transport::Inproc::DirectPipe]
+    # @param recv_queue [SignalingQueue]
+    # @param engine [Engine] for connection_lost / signal_fatal_error callbacks
+    # @param transform [Proc, nil]
+    # @return [Async::Task, nil]
+    #
+    class RecvPump
+      FAIRNESS_MESSAGES = 64
+      FAIRNESS_BYTES    = 1 << 20 # 1 MB
+
+      def self.start(parent_task, conn, recv_queue, engine, transform)
+        if conn.is_a?(Transport::Inproc::DirectPipe) && conn.peer
+          conn.peer.direct_recv_queue     = recv_queue
+          conn.peer.direct_recv_transform = transform
+          return nil
+        end
+
+        if transform
+          parent_task.async(transient: true, annotation: "recv pump") do |task|
+            loop do
+              count = 0
+              bytes = 0
+              while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
+                msg = conn.receive_message
+                msg = transform.call(msg).freeze
+                recv_queue.enqueue(msg)
+                count += 1
+                bytes += msg.is_a?(Array) && msg.first.is_a?(String) ? msg.sum(&:bytesize) : 0
+              end
+              task.yield
+            end
+          rescue Async::Stop
+          rescue Protocol::ZMTP::Error, *CONNECTION_LOST
+            engine.connection_lost(conn)
+          rescue => error
+            engine.signal_fatal_error(error)
+          end
+        else
+          parent_task.async(transient: true, annotation: "recv pump") do |task|
+            loop do
+              count = 0
+              bytes = 0
+              while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
+                msg = conn.receive_message
+                recv_queue.enqueue(msg)
+                count += 1
+                bytes += msg.is_a?(Array) && msg.first.is_a?(String) ? msg.sum(&:bytesize) : 0
+              end
+              task.yield
+            end
+          rescue Async::Stop
+          rescue Protocol::ZMTP::Error, *CONNECTION_LOST
+            engine.connection_lost(conn)
+          rescue => error
+            engine.signal_fatal_error(error)
+          end
+        end
+      end
+    end
+  end
+end