omq 0.12.0 → 0.14.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d5ea3ce51d24a9181cfe7a4c1f2680ed57c5dbba078010a54de5b1c13d27d484
-  data.tar.gz: 97f7e7fcb27a250e71af4732ff06298e7311af052a9d9a6b3868d73059373174
+  metadata.gz: 997f3d83d6ddc56d44341f69743f4f30b9e122f046718246de78da75969ec6fa
+  data.tar.gz: 65f4f071d952c477630bfae4c25f136abfa79350099b58b1be4ca79ec500f2e3
 SHA512:
-  metadata.gz: 9f60b7d45d1b3783b2bde482a6f826ec3b89bfff6a9532836dbd2a8a22622709e7c298cc7c2abae6e596641973636b8972efa92de7258bd1e4164ff62d782237
-  data.tar.gz: 7024b97bb46c9005b73bc7959cc4138bd5498a514af26253441613aa05206d357d1fbd002afff5820c9a3baff0c4351d66bcf1aa738955783272a2c2b459a697
+  metadata.gz: d92503d56e3106987bda522497f42630d7999bca8ba1ddd5190129d0dc567c5dac1714a8f8566601ac1fa309d0e142f87936342667b36588f81e39f785a2b502
+  data.tar.gz: bed4e86b9404a7ecc175c50ba85a2b7ad8daaea1490c8f39a449c8f92c197b039326d73708279838c1678a29235c7a5ff30c1bf36903306d452dccd743059b76

data/CHANGELOG.md

@@ -1,9 +1,92 @@
 # Changelog
 
-## Unreleased
+## 0.14.0 — 2026-04-07
+
+- **Fix recv pump crash with connection wrappers** — `start_direct` called
+  `msg.sum(&:bytesize)` unconditionally, crashing when a `connection_wrapper`
+  (e.g. omq-ractor's `MarshalConnection`) returns deserialized Ruby objects.
+  Byte counting now uses `conn.instance_of?(Protocol::ZMTP::Connection)` to
+  skip non-ZMTP connections (inproc, Ractor bridges).
+- Remove TLS transport dependency from Gemfile.
+- YARD documentation on all public methods and classes.
+- Code style: expand `else X` one-liners, enforce two blank lines between
+  methods and constants.
+- Benchmarks: add per-run timeout (default 30s, `OMQ_BENCH_TIMEOUT` env var)
+  and abort if a group produces no results.
+
+- Add `Engine::Maintenance` — spawns a periodic `Async::Loop.quantized` timer
+  that calls the mechanism's `#maintenance` callback (if defined). Enables
+  automatic cookie key rotation for CurveZMQ and BLAKE3ZMQ server mechanisms.
+- **YJIT: remove redundant `is_a?` guards in recv pump** — the non-transform
+  branch no longer type-checks every message; `conn.receive_message` always
+  returns `Array<String>`.
+- **YJIT: `FanOut#subscribed?` fast path for subscribe-all** — connections
+  subscribed to `""` are tracked in a `@subscribe_all` Set, short-circuiting
+  the per-message prefix scan with an O(1) lookup.
+- **YJIT: remove safe navigation in hot enqueue paths** — `&.enqueue` calls
+  in `FanOut#fan_out_enqueue` and `RoundRobin#enqueue_round_robin` replaced
+  with direct calls; queues are guaranteed to exist for live connections.
+- **Fix PUB/SUB fan-out over inproc and IPC** — restore `respond_to?(:write_wire)`
+  guard in `FanOut#start_conn_send_pump` so DirectPipe connections use
+  `#write_message` instead of the wire-optimized path. Add `DirectPipe#encrypted?`
+  (returns `false`) for the mechanism query.
+- **Code audit: never-instantiated classes** — `RecvPump`, `ConnectionSetup`,
+  and `Reconnect` refactored from class-method namespaces to proper instances
+  that capture shared state. `Heartbeat`, `Maintenance`, and `ConnSendPump`
+  changed from classes to modules (single `self.` method, never instantiated).
+
+## 0.13.0
+
+### Changed
+
+- **`Engine` internals: `ConnectionRecord` + lifecycle state** — three parallel
+  per-connection ivars (`@connections` Array, `@connection_endpoints`,
+  `@connection_promises`) replaced by a single `@connections` Hash keyed by
+  connection, with values `ConnectionRecord = Data.define(:endpoint, :done)`.
+  `@connected_endpoints` renamed to `@dialed` (`Set`). `@closed`/`@closing`
+  booleans replaced by a `@state` symbol (`:open`/`:closing`/`:closed`).
+  Net: −4 instance variables.
+- **`@connections` in `FanOut`, `Sub`, `XSub` routing strategies changed from
+  `Array` to `Set`** — O(1) `#delete` on peer disconnect; semantics already
+  required uniqueness.
+
+### Fixed
+
+- **FanOut send queues no longer drop messages** — per-connection send queues in
+  `FanOut` (PUB/XPUB/RADIO) used `DropQueue` (`Thread::SizedQueue`) which never
+  blocked the publisher fiber. When burst-sending beyond `send_hwm`, the sender
+  ran without yielding and messages were silently dropped. Switched to
+  `Async::LimitedQueue` (`:block`) so the publisher yields when a per-connection
+  queue is full, giving the send pump fiber a chance to drain it.
+
+### Changed
+
+- **Benchmark suite redesign** — replaced ASCII plots (unicode_plot) with JSONL
+  result storage and a colored terminal regression report. Results are appended
+  to `bench/results.jsonl` (gitignored, machine-local). New commands:
+  `ruby bench/run_all.rb` (run all patterns), `ruby bench/report.rb` (compare
+  last runs, highlight regressions/improvements).
 
 ### Added
 
+- **Per-peer HWM** — send and receive high-water marks now apply per connected
+  peer (RFC 28/29/30). Each peer gets its own bounded send queue and its own
+  bounded recv queue. A slow or muted peer no longer steals capacity from
+  other peers. `FairQueue` + `SignalingQueue` aggregate per-connection recv
+  queues with fair round-robin delivery; `RoundRobin` and `FanOut` mixins
+  maintain per-connection send queues with dedicated send pump fibers.
+  `PUSH`/`DEALER`/`PAIR` buffer messages in a staging queue when no peers are
+  connected yet, draining into the first peer's queue on connect.
+- **`FairQueue`** — new aggregator class (`lib/omq/routing/fair_queue.rb`)
+  that fair-queues across per-connection bounded queues. Pending messages from
+  a disconnected peer are drained before the queue is discarded.
+- **`Socket.bind` / `Socket.connect` class-method fix** — now pass the
+  endpoint via `@`/`>` prefix into the constructor so any post-attach
+  initialization in subclasses (e.g. XSUB's `subscribe:` kwarg) runs after
+  the connection is established.
+
+
+
 - **QoS infrastructure** — `Options#qos` attribute (default 0) and inproc
   command queue support for QoS-enabled connections. The
   [omq-qos](https://github.com/paddor/omq-qos) gem activates delivery

data/README.md

@@ -197,6 +197,33 @@ bundle install
 bundle exec rake
 ```
 
+### Full development setup
+
+Set `OMQ_DEV=1` to tell Bundler to load sibling projects from source
+(protocol-zmtp, nuckle, omq-rfc-\*, etc.) instead of released gems.
+This is required for running benchmarks and for testing changes across
+the stack.
+
+```sh
+# clone OMQ and its sibling repos into the same parent directory
+git clone https://github.com/paddor/omq.git
+git clone https://github.com/paddor/protocol-zmtp.git
+git clone https://github.com/paddor/nuckle.git
+git clone https://github.com/paddor/omq-rfc-blake3zmq.git
+git clone https://github.com/paddor/omq-rfc-channel.git
+git clone https://github.com/paddor/omq-rfc-clientserver.git
+git clone https://github.com/paddor/omq-rfc-p2p.git
+git clone https://github.com/paddor/omq-rfc-qos.git
+git clone https://github.com/paddor/omq-rfc-radiodish.git
+git clone https://github.com/paddor/omq-rfc-scattergather.git
+git clone https://github.com/paddor/omq-ffi.git
+git clone https://github.com/paddor/omq-ractor.git
+
+cd omq
+OMQ_DEV=1 bundle install
+OMQ_DEV=1 bundle exec rake
+```
+
 ## License
 
 [ISC](LICENSE)

data/lib/omq/drop_queue.rb

@@ -18,6 +18,7 @@ module OMQ
       @strategy = strategy
     end
 
+
     # Enqueues an item. Drops according to the configured strategy if full.
     #
     # @param item [Object]
@@ -33,6 +34,7 @@ module OMQ
       retry
     end
 
+
     # Removes and returns the next item, blocking if empty.
     #
     # @return [Object]
@@ -45,6 +47,7 @@ module OMQ
       end
     end
 
+
     # @return [Boolean]
     #
     def empty?

data/lib/omq/engine/connection_setup.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Performs ZMTP handshake and registers a new connection.
+    #
+    class ConnectionSetup
+      # @param io [#read, #write, #close] underlying transport stream
+      # @param engine [Engine]
+      # @param as_server [Boolean]
+      # @param endpoint [String, nil]
+      # @param done [Async::Promise, nil] resolved when connection is lost
+      # @return [Connection]
+      #
+      def self.run(io, engine, as_server:, endpoint: nil, done: nil)
+        new(engine).run(io, as_server: as_server, endpoint: endpoint, done: done)
+      end
+
+
+      # @param engine [Engine]
+      #
+      def initialize(engine)
+        @engine = engine
+      end
+
+
+      # Performs the ZMTP handshake, starts heartbeat, and registers the connection.
+      #
+      # @param io [#read, #write, #close]
+      # @param as_server [Boolean]
+      # @param endpoint [String, nil]
+      # @param done [Async::Promise, nil] resolved when connection is lost
+      # @return [Connection]
+      #
+      def run(io, as_server:, endpoint: nil, done: nil)
+        conn = build_connection(io, as_server)
+        conn.handshake!
+        Heartbeat.start(@engine.parent_task, conn, @engine.options, @engine.tasks)
+        conn = @engine.connection_wrapper.call(conn) if @engine.connection_wrapper
+        register(conn, endpoint, done)
+        @engine.emit_monitor_event(:handshake_succeeded, endpoint: endpoint)
+        conn
+      rescue Protocol::ZMTP::Error, *CONNECTION_LOST => error
+        @engine.emit_monitor_event(:handshake_failed, endpoint: endpoint, detail: { error: error })
+        conn&.close
+        raise
+      end
+
+      private
+
+      def build_connection(io, as_server)
+        Protocol::ZMTP::Connection.new(
+          io,
+          socket_type:      @engine.socket_type.to_s,
+          identity:         @engine.options.identity,
+          as_server:        as_server,
+          mechanism:        @engine.options.mechanism&.dup,
+          max_message_size: @engine.options.max_message_size,
+        )
+      end
+
+
+      def register(conn, endpoint, done)
+        @engine.connections[conn] = Engine::ConnectionRecord.new(endpoint: endpoint, done: done)
+        @engine.routing.connection_added(conn)
+        @engine.peer_connected.resolve(conn)
+      end
+    end
+  end
+end

data/lib/omq/engine/heartbeat.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Spawns a heartbeat task for a connection.
+    #
+    # Sends PING frames at +interval+ seconds and closes the connection
+    # if no traffic is seen within +timeout+ seconds.
+    #
+    module Heartbeat
+      # @param parent_task [Async::Task]
+      # @param conn [Connection]
+      # @param options [Options]
+      # @param tasks [Array]
+      #
+      def self.start(parent_task, conn, options, tasks)
+        interval = options.heartbeat_interval
+        return unless interval
+
+        ttl     = options.heartbeat_ttl     || interval
+        timeout = options.heartbeat_timeout || interval
+        conn.touch_heartbeat
+
+        tasks << parent_task.async(transient: true, annotation: "heartbeat") do
+          loop do
+            sleep interval
+            conn.send_command(Protocol::ZMTP::Codec::Command.ping(ttl: ttl, context: "".b))
+            if conn.heartbeat_expired?(timeout)
+              conn.close
+              break
+            end
+          end
+        rescue Async::Stop
+        rescue *CONNECTION_LOST
+          # connection closed
+        end
+      end
+    end
+  end
+end

data/lib/omq/engine/maintenance.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+require "async/loop"
+
+module OMQ
+  class Engine
+    # Spawns a periodic maintenance task for the parent mechanism.
+    #
+    # The mechanism declares maintenance needs via +#maintenance+,
+    # which returns +{ interval:, task: }+ or nil.
+    #
+    module Maintenance
+      # @param parent_task [Async::Task]
+      # @param mechanism [#maintenance, nil]
+      # @param tasks [Array<Async::Task>]
+      #
+      def self.start(parent_task, mechanism, tasks)
+        return unless mechanism.respond_to?(:maintenance)
+        spec = mechanism.maintenance
+        return unless spec
+
+        interval = spec[:interval]
+        callable = spec[:task]
+
+        tasks << parent_task.async(transient: true, annotation: "mechanism maintenance") do
+          Async::Loop.quantized(interval: interval) do
+            callable.call
+          end
+        rescue Async::Stop
+          # clean shutdown
+        end
+      end
+    end
+  end
+end

data/lib/omq/engine/reconnect.rb

@@ -0,0 +1,82 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Schedules reconnect attempts with exponential back-off.
+    #
+    # Runs a background task that loops until a connection is established
+    # or the engine is closed.
+    #
+    class Reconnect
+      # @param endpoint [String]
+      # @param options [Options]
+      # @param parent_task [Async::Task]
+      # @param engine [Engine]
+      # @param delay [Numeric, nil] initial delay (defaults to reconnect_interval)
+      #
+      def self.schedule(endpoint, options, parent_task, engine, delay: nil)
+        new(engine, endpoint, options).run(parent_task, delay: delay)
+      end
+
+
+      # @param engine [Engine]
+      # @param endpoint [String]
+      # @param options [Options]
+      #
+      def initialize(engine, endpoint, options)
+        @engine   = engine
+        @endpoint = endpoint
+        @options  = options
+      end
+
+
+      # Spawns a background task that retries the connection with exponential backoff.
+      #
+      # @param parent_task [Async::Task]
+      # @param delay [Numeric, nil] initial delay override
+      # @return [void]
+      #
+      def run(parent_task, delay: nil)
+        delay, max_delay = init_delay(delay)
+
+        @engine.tasks << parent_task.async(transient: true, annotation: "reconnect #{@endpoint}") do
+          loop do
+            break if @engine.closed?
+            sleep delay if delay > 0
+            break if @engine.closed?
+            begin
+              @engine.transport_for(@endpoint).connect(@endpoint, @engine)
+              break
+            rescue *CONNECTION_LOST, *CONNECTION_FAILED, Protocol::ZMTP::Error
+              delay = next_delay(delay, max_delay)
+              @engine.emit_monitor_event(:connect_retried, endpoint: @endpoint, detail: { interval: delay })
+            end
+          end
+        rescue Async::Stop
+        rescue => error
+          @engine.signal_fatal_error(error)
+        end
+      end
+
+      private
+
+      def init_delay(delay)
+        ri = @options.reconnect_interval
+        if ri.is_a?(Range)
+          [delay || ri.begin, ri.end]
+        else
+          [delay || ri, nil]
+        end
+      end
+
+
+      def next_delay(delay, max_delay)
+        ri    = @options.reconnect_interval
+        delay = delay * 2
+        delay = [delay, max_delay].min if max_delay
+        delay = (ri.is_a?(Range) ? ri.begin : ri) if delay == 0
+        delay
+      end
+    end
+  end
+end

data/lib/omq/engine/recv_pump.rb

@@ -0,0 +1,119 @@
+# frozen_string_literal: true
+
+module OMQ
+  class Engine
+    # Recv pump for a connection.
+    #
+    # For inproc DirectPipe: wires the direct recv path (no fiber spawned).
+    # For TCP/IPC: spawns a transient task that reads messages from the
+    # connection and enqueues them into +recv_queue+.
+    #
+    # The two-method structure (with/without transform) is intentional for
+    # YJIT: it gives the JIT a monomorphic call per routing strategy instead
+    # of a megamorphic `transform.call` dispatch inside a shared loop.
+    #
+    class RecvPump
+      FAIRNESS_MESSAGES = 64
+      FAIRNESS_BYTES    = 1 << 20 # 1 MB
+
+
+      # Public entry point — callers use the class method.
+      #
+      # @param parent_task [Async::Task]
+      # @param conn [Connection, Transport::Inproc::DirectPipe]
+      # @param recv_queue [SignalingQueue]
+      # @param engine [Engine]
+      # @param transform [Proc, nil]
+      # @return [Async::Task, nil]
+      #
+      def self.start(parent_task, conn, recv_queue, engine, transform)
+        new(conn, recv_queue, engine).start(parent_task, transform)
+      end
+
+
+      # @param conn [Connection, Transport::Inproc::DirectPipe]
+      # @param recv_queue [Routing::SignalingQueue]
+      # @param engine [Engine]
+      #
+      def initialize(conn, recv_queue, engine)
+        @conn        = conn
+        @recv_queue  = recv_queue
+        @engine      = engine
+        @count_bytes = conn.instance_of?(Protocol::ZMTP::Connection)
+      end
+
+
+      # Starts the recv pump. For inproc DirectPipe, wires the direct path
+      # (no task spawned). For TCP/IPC, spawns a fiber that reads messages.
+      #
+      # @param parent_task [Async::Task]
+      # @param transform [Proc, nil] optional per-message transform
+      # @return [Async::Task, nil]
+      #
+      def start(parent_task, transform)
+        if @conn.is_a?(Transport::Inproc::DirectPipe) && @conn.peer
+          @conn.peer.direct_recv_queue     = @recv_queue
+          @conn.peer.direct_recv_transform = transform
+          return nil
+        end
+
+        if transform
+          start_with_transform(parent_task, transform)
+        else
+          start_direct(parent_task)
+        end
+      end
+
+      private
+
+
+      def start_with_transform(parent_task, transform)
+        conn, recv_queue, count_bytes = @conn, @recv_queue, @count_bytes
+
+        parent_task.async(transient: true, annotation: "recv pump") do |task|
+          loop do
+            count = 0
+            bytes = 0
+            while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
+              msg = conn.receive_message
+              msg = transform.call(msg).freeze
+              recv_queue.enqueue(msg)
+              count += 1
+              bytes += msg.sum(&:bytesize) if count_bytes
+            end
+            task.yield
+          end
+        rescue Async::Stop
+        rescue Protocol::ZMTP::Error, *CONNECTION_LOST
+          @engine.connection_lost(conn)
+        rescue => error
+          @engine.signal_fatal_error(error)
+        end
+      end
+
+
+      def start_direct(parent_task)
+        conn, recv_queue, count_bytes = @conn, @recv_queue, @count_bytes
+
+        parent_task.async(transient: true, annotation: "recv pump") do |task|
+          loop do
+            count = 0
+            bytes = 0
+            while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
+              msg = conn.receive_message
+              recv_queue.enqueue(msg)
+              count += 1
+              bytes += msg.sum(&:bytesize) if count_bytes
+            end
+            task.yield
+          end
+        rescue Async::Stop
+        rescue Protocol::ZMTP::Error, *CONNECTION_LOST
+          @engine.connection_lost(conn)
+        rescue => error
+          @engine.signal_fatal_error(error)
+        end
+      end
+    end
+  end
+end