omq 0.23.0 → 0.26.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2a0e72756aae2179b8c790d5b65c54a82b516a17d11f760187a6fefecf6bf2c1
-  data.tar.gz: d6ce860b558977939b28c3d1f23d7bfc0740cf602fb63a519107a59cdca3c8c3
+  metadata.gz: c2c1793056ec0f8ed585034cf19a939a7bb22278d865dc9f570a2117e1bd45d4
+  data.tar.gz: b8502a9780715a9e565b52b987ad1a698c07ff76c08c1a6e38f01b833b724f71
 SHA512:
-  metadata.gz: fa8a7c98147aa4f0fea5fad3004e8a964105a2005ed645d03b0fd1f0fba67bfc4952a3f4d14926fbaf4293efdcada93235b2ba59956737ef0ced5c1b97d35ea2
-  data.tar.gz: b5878650300c7135e04ec8fffcadef6d5101f77c8f2a7e3e6c1407faebe57714b613398c3fecf71b561cb8bdb6c25a70646d812c395db46f78827e97f796c8e7
+  metadata.gz: f3cbc43a10f74f0c1a872071871b9d5daddae41f2304823f8b57388a5c0c99cfb52e9a4118d9504c462d609af759c86a290d3235867fa5d22d46b9ffda63203c
+  data.tar.gz: 44ca650e5c321a56fd4649cbebe0a4bd60efc3127547a5da0010582ed0fbb42fe4bcbf705ddfbf44596266a1d37869d49d4bef03df08551b703091d733b4e10f

data/CHANGELOG.md

@@ -1,5 +1,145 @@
 # Changelog
 
+## 0.26.2 — 2026-04-20
+
+### Fixed
+
+- **Ruby 3.3 compatibility.** Replaced bare `it` block references
+  with explicit block parameters in `engine/recv_pump.rb` and
+  `writable.rb`. Ruby 3.3 warned that `it` would change meaning in
+  3.4; the explicit params work on both.
+
+## 0.26.1 — 2026-04-20
+
+### Fixed
+
+- **Inproc Pipe: tolerate non-String parts.** The BINARY-encoding
+  upgrade introduced in 0.25 called `.encoding` on every frame,
+  crashing when plugins (e.g. omq-ractor's `ShareableConnection`)
+  carried arbitrary Ruby objects through inproc. Non-String parts
+  now pass through untouched; String parts still get the
+  frozen-string-literal → BINARY upgrade.
+
+## 0.26.0 — 2026-04-20
+
+### Added
+
+- **FFI backend absorbed in-tree.** The libzmq-backed `OMQ::FFI::Engine`
+  (previously shipped as the separate `omq-ffi` gem) now lives in
+  `lib/omq/ffi/`. Load with `require "omq/ffi"` and select per socket
+  via `OMQ::PUSH.new(backend: :ffi)`. `lib/omq/socket.rb` also
+  lazy-requires `omq/ffi` on first `:ffi` use, so the explicit require
+  is optional.
+- **Auto-running FFI interop tests.** `test/omq/ffi_test.rb` (FFI
+  backend) and `test/omq/interop_test.rb` (FFI ↔ pure Ruby wire
+  compatibility) now run as part of `rake test` whenever the `ffi` gem
+  and system libzmq are both available. They self-skip otherwise —
+  detection runs once in `test/test_helper.rb` as `OMQ_FFI_AVAILABLE`.
+
+### Notes
+
+- `ffi` remains optional and is NOT a runtime dependency of `omq`.
+  Install it explicitly (`gem install ffi` + system libzmq 4.x) to use
+  the `:ffi` backend. The omq-ffi gem is superseded; existing pins to
+  `omq-ffi ~> 0.3` keep working via its own dependency on `omq ~> 0.23`.
+
+## 0.25.0 — 2026-04-20
+
+### Added
+
+- **Recv-pump transforms can drop messages.** A `transform` block passed
+  to `Engine#start_recv_pump` may now return `nil` to discard the
+  message instead of enqueueing it to the application's recv queue.
+  The pump still counts the dropped message toward its per-connection
+  fairness caps (64 msgs / 1 MiB), so a duplicate flood can't starve
+  siblings. omq-qos 0.3.0 uses this at QoS >= 2 for dedup-set hits:
+  the transform ACKs the sender and returns `nil`.
+
+### Changed
+
+- **Uniform frozen + BINARY contract on both sides of the wire —
+  restoring pre-0.24 behavior.** 0.24 dropped freezing from the
+  send/receive paths to chase throughput numbers, which left inproc
+  with an unsafe shared-reference contract (sender and receiver share
+  the same array and strings) and made the contract differ by
+  transport. Safety is back, minus the `.b` copy that was the
+  actually-expensive part of the old path. Invariants:
+
+  - `Writable#send` freezes every part (and the parts array, if one
+    was passed). Unfrozen non-BINARY parts are re-tagged to
+    `Encoding::BINARY` in place — a flag flip, no allocation.
+  - Receivers always get frozen `BINARY`-tagged parts. TCP/IPC get
+    this via byteslice on the wire + recv-pump freeze. Inproc gets
+    it via `Pipe#send_message`, which only allocates (one `.b` copy
+    per part) in the pathological case of a frozen non-BINARY part
+    — the typical `# frozen_string_literal: true` UTF-8 literal.
+
+  Mutation bugs surface as `FrozenError` instead of silently
+  corrupting a shared reference on inproc. Cost on inproc is ~20-30%
+  throughput; TCP/IPC unaffected.
+
+- **String-like part coercion via `#to_str`.** Non-String parts are
+  coerced via `#to_str` (not `#to_s`) — an object must be explicitly
+  string-like to serialize. Passing `42`, `:foo`, or `nil` raises
+  `NoMethodError` instead of silently accepting a `#to_s`
+  representation or producing a zero-byte frame from a `nil`. Use
+  `""` to send an empty frame.
+
+- **Inproc `needs_commands?` accepts nilable `options.qos`.** Core
+  `Options#qos` is still an Integer (default `0`), but omq-qos 0.3
+  stores either `nil` (QoS 0) or an `OMQ::QoS` instance (levels 1–3)
+  in that slot. The inproc transport's command-queue decision now
+  treats both Integer `0` and `nil` as disabled; any non-zero
+  Integer or non-nil object forces the command-queue path.
+
+
+## 0.24.0 — 2026-04-18
+
+### Changed
+
+- **Caller owns message parts.** `Writable#send` no longer deep-freezes or
+  binary-coerces the caller's input. The contract is now libzmq-style:
+  don't mutate parts after sending. `#receive` likewise returns mutable
+  arrays of mutable strings. This removes a full-payload allocation per
+  message (`.b.freeze`) on the send path and a per-frame freeze on the
+  receive path.
+
+- **No more implicit `#to_s` / nil coercion.** Passing a non-string part
+  (e.g. Integer, Symbol, nil) will raise `NoMethodError` at the wire layer
+  instead of being silently converted. The `EMPTY_PART` constant is gone.
+
+- **Reactor fast path for `#send` / `#receive`.** When the socket was
+  bound/connected from an Async fiber, hot-path I/O skips `Reactor.run`
+  entirely and calls the engine directly (with an `Async::Task#with_timeout`
+  wrapper only when a timeout is configured). The shared IO thread is used
+  only when the socket was created from a non-Async thread.
+
+### Performance
+
+Combined effect of caller-owns-data + Reactor fast path on inproc:
+
+- PUSH/PULL inproc 1-peer: **+105% to +128%** msg/s across payload sizes
+- PUSH/PULL inproc 3-peer: **+63% to +111%** msg/s
+- PUSH/PULL ipc: +5% to +17%
+- TCP numbers unchanged (OS/syscall-dominated)
+
+### Removed
+
+- `Writable#freeze_message` and `#frozen_binary` private helpers.
+- `Writable::EMPTY_PART` constant.
+
+
+## 0.23.1 — 2026-04-18
+
+### Fixed
+
+- **SCATTER double-tracked each peer.** `Routing::Scatter#connection_added`
+  appended to `@connections` and then called `add_round_robin_send_connection`,
+  which appends again — so every connected peer had two entries in the list.
+  `#connection_removed` deleted only one on disconnect, leaving a stale entry
+  behind. Fixed by dropping the duplicate append.
+
+
 ## 0.23.0 — 2026-04-17
 
 ### Added
@@ -997,9 +1137,10 @@
 ### Added
 
 - **`backend:` kwarg** — all socket types accept `backend: :ffi` to use
-  the libzmq FFI backend (via the [omq-ffi](https://github.com/paddor/omq-ffi)
-  gem). Default is `:ruby` (pure Ruby ZMTP). Enables interop testing and
-  access to libzmq-specific features without changing the socket API.
+  the libzmq FFI backend (then shipped separately as the `omq-ffi` gem;
+  absorbed in-tree in 0.26.0). Default is `:ruby` (pure Ruby ZMTP).
+  Enables interop testing and access to libzmq-specific features without
+  changing the socket API.
 - **TLS transport (`tls+tcp://`)** — TLS v1.3 on top of TCP using Ruby's
   stdlib `openssl`. Set `socket.tls_context` to an `OpenSSL::SSL::SSLContext`
   before bind/connect. Per-socket (not per-endpoint), frozen on first use.

data/README.md

@@ -5,9 +5,9 @@
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
 [![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
 
-> **1.23M msg/s** inproc | **361k msg/s** ipc | **358k msg/s** tcp
+> **1.64M msg/s** inproc | **294k msg/s** ipc | **308k msg/s** tcp
 >
-> **9.1 µs** inproc latency | **49 µs** ipc | **64 µs** tcp
+> **8.7 µs** inproc latency | **51 µs** ipc | **64 µs** tcp
 >
 > Ruby 4.0 + YJIT on a Linux VM — see [`bench/`](bench/) for full results
 
@@ -162,6 +162,12 @@ All sockets are thread-safe. Default HWM is 1000 messages per socket.
 frames cause the connection to be dropped before the body is read from the
 wire. Classes live under `OMQ::` (alias: `ØMQ`).
 
+**Received messages are frozen** across all transports (inproc, ipc, tcp).
+The array returned by `#receive` and every part inside it is frozen —
+mutating a received part raises `FrozenError` rather than silently
+corrupting a shared reference on the inproc fast path. `dup` a part if
+you need to mutate it.
+
 #### Standard (multipart messages)
 
 | Pattern | Send | Receive | When HWM full |
@@ -207,12 +213,23 @@ echo "hello" | omq req -c tcp://localhost:5555
 
 See the [omq-cli README](https://github.com/paddor/omq-cli) for full documentation.
 
+## Optional libzmq backend
+
+OMQ ships with an optional libzmq FFI backend. Same socket API, but
+backed by libzmq instead of the pure Ruby ZMTP stack. Useful when you
+need libzmq-specific features or for verifying wire compatibility.
+
+```ruby
+require "omq/ffi"
+push = OMQ::PUSH.new(backend: :ffi)
+```
+
+Requires the `ffi` gem and a system libzmq 4.x. `ffi` is not a runtime
+dependency of `omq` — install it explicitly (`gem install ffi`) if you
+want the `:ffi` backend.
+
 ## Companion Gems
 
-- **[omq-ffi](https://github.com/paddor/omq-ffi)** — libzmq FFI backend. Same
-  OMQ socket API, but backed by libzmq instead of the pure Ruby ZMTP stack.
-  Useful for interop testing and when you need libzmq-specific features.
-  Requires libzmq installed.
 - **[omq-ractor](https://github.com/paddor/omq-ractor)** — bridge OMQ sockets
   into Ruby Ractors for true parallel processing across cores. I/O stays on the
   main Ractor, worker Ractors do pure computation.
@@ -244,7 +261,6 @@ the stack.
 git clone https://github.com/paddor/omq.git
 git clone https://github.com/paddor/protocol-zmtp.git
 git clone https://github.com/paddor/omq-zstd.git
-git clone https://github.com/paddor/omq-ffi.git
 git clone https://github.com/paddor/omq-ractor.git
 git clone https://github.com/paddor/nuckle.git
 

data/lib/omq/client_server.rb

@@ -60,7 +60,7 @@ module OMQ
     # @return [self]
     #
     def send_to(routing_id, message)
-      parts = [routing_id.b.freeze, message.b.freeze]
+      parts = [routing_id, message]
       Reactor.run(timeout: @options.write_timeout) { @engine.enqueue_send(parts) }
       self
     end

data/lib/omq/engine/connection_lifecycle.rb

@@ -5,7 +5,7 @@ module OMQ
     # Owns the full arc of *one* connection: handshake → ready → closed.
     #
     # Scope boundary: ConnectionLifecycle tracks a single peer link
-    # (one ZMTP connection or one inproc DirectPipe). SocketLifecycle
+    # (one ZMTP connection or one inproc Pipe). SocketLifecycle
     # owns the socket-wide state above it — first-peer/last-peer
     # signaling, reconnect enable flag, the parent task tree, and the
     # open → closing → closed transitions that gate close-time drain.
@@ -43,7 +43,7 @@ module OMQ
       }.freeze
 
 
-      # @return [Protocol::ZMTP::Connection, Transport::Inproc::DirectPipe, nil]
+      # @return [Protocol::ZMTP::Connection, Transport::Inproc::Pipe, nil]
       attr_reader :conn
 
 
@@ -120,9 +120,9 @@ module OMQ
 
 
       # Registers an already-connected inproc pipe as :ready.
-      # No handshake — inproc DirectPipe bypasses ZMTP entirely.
+      # No handshake — inproc Pipe bypasses ZMTP entirely.
       #
-      # @param pipe [Transport::Inproc::DirectPipe]
+      # @param pipe [Transport::Inproc::Pipe]
       #
       def ready_direct!(pipe)
         ready!(pipe)
@@ -161,7 +161,9 @@ module OMQ
 
 
       def ready!(conn)
-        conn = @engine.connection_wrapper.call(conn) if @engine.connection_wrapper
+        if @engine.connection_wrapper
+          conn = @engine.connection_wrapper.call(conn)
+        end
 
         if @endpoint
           transport_obj = @engine.transport_object_for(@endpoint)
@@ -177,7 +179,7 @@ module OMQ
         @engine.peer_connected.resolve(@conn)
         transition!(:ready)
 
-        # No supervisor if nothing to supervise: inproc DirectPipes
+        # No supervisor if nothing to supervise: inproc Pipes
         # wire the recv/send paths synchronously (no task-based pumps),
         # and isolated unit tests use a FakeEngine without pumps at all.
         # Waiting on an empty barrier returns immediately and would
@@ -213,6 +215,7 @@ module OMQ
 
       def tear_down!(reconnect:, reason: nil)
         return if @state == :closed
+
         transition!(:closed)
         @engine.connections.delete(@conn)
         @engine.routing.connection_removed(@conn) if @conn
@@ -244,11 +247,14 @@ module OMQ
 
       def transition!(new_state)
         allowed = TRANSITIONS[@state]
+
         unless allowed&.include?(new_state)
           raise InvalidTransition, "#{@state} → #{new_state}"
         end
+
         @state = new_state
       end
+
     end
   end
 end

data/lib/omq/engine/heartbeat.rb

@@ -9,7 +9,7 @@ module OMQ
     #
     module Heartbeat
       # @param parent [Async::Task, Async::Barrier] parent to spawn under
-      # @param conn [Connection]
+      # @param conn [Protocol::ZMTP::Connection]
       # @param options [Options]
       #
       def self.start(parent, conn, options)

data/lib/omq/engine/recv_pump.rb

@@ -4,7 +4,7 @@ module OMQ
   class Engine
     # Recv pump for a connection.
     #
-    # For inproc DirectPipe: wires the direct recv path (no fiber spawned).
+    # For inproc Pipe: wires the direct recv path (no fiber spawned).
     # For TCP/IPC: spawns a transient task that reads messages from the
     # connection and enqueues them into +recv_queue+.
     #
@@ -29,7 +29,7 @@ module OMQ
       # Public entry point — callers use the class method.
       #
       # @param parent [Async::Task, Async::Barrier] parent to spawn under
-      # @param conn [Connection, Transport::Inproc::DirectPipe]
+      # @param conn [Protocol::ZMTP::Connection, Transport::Inproc::Pipe]
       # @param recv_queue [Async::LimitedQueue]
       # @param engine [Engine]
       # @param transform [Proc, nil]
@@ -40,7 +40,7 @@ module OMQ
       end
 
 
-      # @param conn [Connection, Transport::Inproc::DirectPipe]
+      # @param conn [Protocol::ZMTP::Connection, Transport::Inproc::Pipe]
       # @param recv_queue [Async::LimitedQueue]
       # @param engine [Engine]
       #
@@ -52,7 +52,7 @@ module OMQ
       end
 
 
-      # Starts the recv pump. For inproc DirectPipe, wires the direct path
+      # Starts the recv pump. For inproc Pipe, wires the direct path
       # (no task spawned). For TCP/IPC, spawns a fiber that reads messages.
       #
       # @param parent_task [Async::Task]
@@ -60,7 +60,7 @@ module OMQ
       # @return [Async::Task, nil]
       #
       def start(parent_task, transform)
-        if @conn.is_a?(Transport::Inproc::DirectPipe) && @conn.peer
+        if @conn.is_a?(Transport::Inproc::Pipe) && @conn.peer
           @conn.peer.wire_direct_recv(@recv_queue, transform)
           return nil
         end
@@ -72,6 +72,7 @@ module OMQ
         end
       end
 
+
       private
 
 
@@ -79,6 +80,12 @@ module OMQ
       # cross-Ractor transport). Kept separate from {#start_direct} so
       # YJIT sees a monomorphic transform.call site.
       #
+      # A transform that returns +nil+ drops the message — the recv pump
+      # still counts it toward fairness (so dup-floods can't starve
+      # siblings) but it is neither emitted nor enqueued to the
+      # application. omq-qos uses this at QoS >= 2 for dedup-set hits:
+      # the transform ACKs the sender and returns nil.
+      #
       # @param parent [Async::Task, Async::Barrier]
       # @param transform [Proc]
       # @return [Async::Task]
@@ -93,18 +100,11 @@ module OMQ
 
             while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
               msg = conn.receive_message
-              msg = transform.call(msg).freeze
-
-              # Emit the verbose trace BEFORE enqueueing so the monitor
-              # fiber is woken before the application fiber -- the
-              # async scheduler is FIFO on the ready list, so this
-              # preserves log-before-stdout ordering for -vvv traces.
-              engine.emit_verbose_msg_received(conn, msg)
-              recv_queue.enqueue(msg)
+              msg.each { |part| part.freeze }
+              msg.freeze
 
-              count += 1
-
-              # hot path
+              # hot path bytes — count before transform so dropped
+              # messages still contribute to the fairness cap.
               if count_bytes
                 if msg.size == 1
                   bytes += msg.first.bytesize
@@ -116,6 +116,17 @@ module OMQ
                   end
                 end
               end
+
+              count += 1
+              transformed = transform.call(msg)
+              next unless transformed
+
+              # Emit the verbose trace BEFORE enqueueing so the monitor
+              # fiber is woken before the application fiber -- the
+              # async scheduler is FIFO on the ready list, so this
+              # preserves log-before-stdout ordering for -vvv traces.
+              engine.emit_verbose_msg_received(conn, transformed)
+              recv_queue.enqueue(transformed)
             end
 
             task.yield
@@ -146,6 +157,8 @@ module OMQ
 
             while count < FAIRNESS_MESSAGES && bytes < FAIRNESS_BYTES
               msg = conn.receive_message
+              msg.each { |part| part.freeze }
+              msg.freeze
               engine.emit_verbose_msg_received(conn, msg)
               recv_queue.enqueue(msg)
 

data/lib/omq/engine.rb

@@ -100,6 +100,7 @@ module OMQ
     def parent_task       = @lifecycle.parent_task
     def barrier           = @lifecycle.barrier
     def closed?           = @lifecycle.closed?
+    def on_io_thread?     = @lifecycle.on_io_thread
 
 
     # Enables or disables auto-reconnect for dropped connections.
@@ -280,10 +281,10 @@ module OMQ
     end
 
 
-    # Called by inproc transport with a pre-validated DirectPipe.
+    # Called by inproc transport with a pre-validated Pipe.
     # Skips ZMTP handshake — just registers with routing strategy.
     #
-    # @param pipe [Transport::Inproc::DirectPipe]
+    # @param pipe [Transport::Inproc::Pipe]
     # @return [void]
     #
     def connection_ready(pipe, endpoint: nil)
@@ -328,7 +329,7 @@ module OMQ
 
     # Starts a recv pump for a connection, or wires the inproc fast path.
     #
-    # @param conn [Connection, Transport::Inproc::DirectPipe]
+    # @param conn [Protocol::ZMTP::Connection, Transport::Inproc::Pipe]
     # @param recv_queue [Async::LimitedQueue]
     # @yield [msg] optional per-message transform
     # @return [Async::Task, nil]
@@ -345,7 +346,7 @@ module OMQ
 
     # Called when a connection is lost.
     #
-    # @param connection [Connection]
+    # @param connection [Protocol::ZMTP::Connection]
     # @return [void]
     #
     def connection_lost(connection)
@@ -460,7 +461,7 @@ module OMQ
     # pumps blocked on `dequeue` waiting for messages that will never
     # be written.
     #
-    # @param conn [Connection, Transport::Inproc::DirectPipe]
+    # @param conn [Protocol::ZMTP::Connection, Transport::Inproc::Pipe]
     # @param annotation [String]
     #
     def spawn_conn_pump_task(conn, annotation:, &block)