omq 0.9.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a14af693117ccfb193f54e9bc2a4faae9c2bf6396e7769abee97f279fd62fdba
-  data.tar.gz: 1f5e99ae3b61402d2b9e506c84fdb07d2003695515d3f736159cf2c373cc39d0
+  metadata.gz: b736b6b715f5c4f330d21be278450600b2642239b8bea9e85957f516e89211ed
+  data.tar.gz: 34c057a32055e7333e4299e85c0fee9e798b43ea34d021f237ba6778338b65f1
 SHA512:
-  metadata.gz: 76539e2a9fbdcbe59590e1fa499e300b8f7ad3d527255af6f94665139c3c0a60151f0b82202df372df1cdbc16e90f46369aa7b159342f951cf52babc897b837c
-  data.tar.gz: dba241d3283a0a0330ba0d12081c08f3ae074acaa8be7b1e9292dee9c19a483d822f896a4442ec585005dbeb36efa704b87f51c53e07c1de0ede54a85150f55c
+  metadata.gz: e95a6df6d4eb56ac1fc0586f61cfdb77d0afe9de620d3e540ffbae1256c2fe6526556485215eda3de20c8f34a332f51a68e5bb075087665049c0219bb07f6960
+  data.tar.gz: 8b5c8d38b7523e9c6ab30e17c21b0cf4adb2086e1bdde90298f62e42f4b8f2336cdd95c47293c3fb24fa41f376aed8a2ec6d1121fc56d41aa83d4afdc289d707

data/CHANGELOG.md

@@ -1,5 +1,134 @@
 # Changelog
 
+## 0.11.0
+
+### Added
+
+- **`backend:` kwarg** — all socket types accept `backend: :ffi` to use
+  the libzmq FFI backend (via the [omq-ffi](https://github.com/paddor/omq-ffi)
+  gem). Default is `:ruby` (pure Ruby ZMTP). Enables interop testing and
+  access to libzmq-specific features without changing the socket API.
+- **TLS transport (`tls+tcp://`)** — TLS v1.3 on top of TCP using Ruby's
+  stdlib `openssl`. Set `socket.tls_context` to an `OpenSSL::SSL::SSLContext`
+  before bind/connect. Per-socket (not per-endpoint), frozen on first use.
+  SNI set automatically from the endpoint hostname. Bad TLS handshakes are
+  dropped without killing the accept loop. `OpenSSL::SSL::SSLError` added
+  to `CONNECTION_LOST` for automatic reconnection on TLS failures.
+  Accompanied by a draft RFC (`rfc/zmtp-tls.md`) defining the transport
+  mapping for ZMTP 3.1 over TLS.
+- **PUB/RADIO fan-out pre-encoding** — ZMTP frames are encoded once per
+  message and written as raw wire bytes to all non-CURVE subscribers.
+  Eliminates redundant `Frame.new` + `#to_wire` calls during fan-out.
+  CURVE connections (which encrypt at the ZMTP level) still encode
+  per-connection. TLS, NULL, and PLAIN all benefit since TLS encrypts
+  below ZMTP. Requires protocol-zmtp `Frame.encode_message` and
+  `Connection#write_wire`.
+- **CURVE benchmarks** — all per-pattern benchmarks now include CURVE
+  (via rbnacl) alongside inproc, ipc, tcp, and tls transports.
+- **Engine `connection_wrapper` hook** — optional proc on Engine that wraps
+  new connections (both inproc and tcp/ipc) at creation time. Used by the
+  omq-ractor gem for per-connection serialization (Marshal for tcp/ipc,
+  `Ractor.make_shareable` for inproc).
+- **Queue-style interface** — readable sockets gain `#dequeue(timeout:)`,
+  `#pop`, `#wait`, and `#each`; writable sockets gain `#enqueue` and
+  `#push`. Inspired by `Async::Queue`. `#wait` blocks indefinitely
+  (ignores `read_timeout`); `#each` returns gracefully on timeout.
+- **Recv pump fairness** — each connection yields to the fiber scheduler
+  after 64 messages or 1 MB (whichever comes first). Prevents a fast or
+  large-message connection from starving slower peers when the consumer
+  keeps up. Byte counting gracefully handles non-string messages (e.g.
+  deserialized objects from connection wrappers).
+- **Per-pattern benchmark suite** — `bench/{push_pull,req_rep,router_dealer,dealer_dealer,pub_sub,pair}/omq.rb`
+  with shared helpers (`bench_helper.rb`) and UnicodePlot braille line
+  charts (`plot.rb`). Each benchmark measures throughput (msg/s) and
+  bandwidth (MB/s) across transports (inproc, ipc, tcp, tls, curve),
+  message sizes (64 B–64 KB), and peer counts (1, 3). Plots are written to per-directory
+  `README.md` files for easy diffing across versions.
+
+### Changed
+
+- **SUB/XSUB `prefix:` kwarg renamed to `subscribe:`** — aligns with
+  ZeroMQ conventions. `subscribe: nil` (no subscription) remains the
+  default; pass `subscribe: ''` to subscribe to everything, or
+  `subscribe: 'topic.'` for a prefix filter.
+- **Scenario benchmarks moved to `bench/scenarios/`** — broker,
+  draft_types, flush_batching, hwm_backpressure, large_messages,
+  multiframe, pubsub_fanout, ractors_vs_async, ractors_vs_fork,
+  reconnect_storm, and reqrep_throughput moved from `bench/` top level.
+
+### Removed
+
+- **Old flat benchmarks** — `bench/throughput.rb`, `bench/latency.rb`,
+  `bench/pipeline_mbps.rb`, `bench/run_all.sh` replaced by per-pattern
+  benchmarks.
+- **`bench/cli/`** — CLI-specific benchmarks (fib pipeline, latency,
+  throughput shell scripts) moved to the omq-cli repository.
+
+## 0.10.0 — 2026-04-01
+
+### Added
+
+- **Auto-close sockets via Async task tree** — all engine tasks (accept
+  loops, connection tasks, send/recv pumps, heartbeats, reconnect loops,
+  reapers) now live under the caller's Async task. When the `Async` block
+  exits, tasks are stopped and `ensure` blocks close IO resources.
+  Explicit `Socket#close` is no longer required (but remains available
+  and idempotent).
+- **Non-Async usage** — sockets work outside `Async do…end`. A shared IO
+  thread hosts the task tree; all blocking operations (bind, connect,
+  send, receive, close) are dispatched to it transparently via
+  `Reactor.run`. The IO thread shuts down cleanly at process exit,
+  respecting the longest linger across all sockets.
+- **Recv prefetching** — `#receive` internally drains up to 64 messages
+  per queue dequeue, buffering the excess behind a Mutex. Subsequent
+  calls return from the buffer without touching the queue. Thread-safe
+  on JRuby. TCP 64B pipelined: 30k → 221k msg/s (7x).
+
+### Changed
+
+- **Transports are pure IO** — TCP and IPC transports no longer spawn
+  tasks. They create server sockets and return them; Engine owns the
+  accept loops.
+- **Reactor simplified** — `spawn_pump` and `PumpHandle` removed.
+  Reactor exposes `root_task` (shared IO thread's root Async task)
+  and `run` (cross-thread dispatch). `stop!` respects max linger.
+- **Flatten `OMQ::ZMTP` namespace into `OMQ`** — with the ZMTP protocol
+  layer extracted to `protocol-zmtp`, the `ZMTP` sub-namespace no longer
+  makes sense. Engine, routing, transport, and mixins now live directly
+  under `OMQ::`. Protocol-zmtp types are referenced as `Protocol::ZMTP::*`.
+
+### Performance
+
+- **Direct pipe bypass for single-peer inproc** — PAIR, CHANNEL, and
+  single-peer RoundRobin types (PUSH, REQ, DEALER, CLIENT, SCATTER)
+  enqueue directly into the receiver's recv queue, skipping the
+  send_queue and send pump entirely.
+  Inproc PUSH/PULL: 200k → 980k msg/s (5x).
+- **Uncapped send queue drain** — the send pump drains the entire queue
+  per cycle instead of capping at 64 messages. IO::Stream auto-flushes
+  at 64 KB, so writes hit the wire naturally under load.
+  IPC latency −12%, TCP latency −10%.
+- **Remove `.b` allocations from PUB/SUB subscription matching** —
+  `FanOut#subscribed?` no longer creates temporary binary strings per
+  comparison; both topic and prefix are guaranteed binary at rest.
+- **Reuse `written` Set and `latest` Hash across batches** in all send
+  pumps (fan-out, round-robin, router, server, peer, rep, radio),
+  eliminating per-batch object allocation.
+- **O(1) `connection_removed` for identity-routed sockets** — Router,
+  Server, and Peer now maintain a reverse index instead of scanning.
+- **`freeze_message` fast path** — skip `.b.freeze` when the string is
+  already a frozen binary string.
+- **Pre-frozen empty frame constants** for REQ/REP delimiter frames.
+
+### Fixed
+
+- **Reapers no longer crash on inproc DirectPipe** — PUSH and SCATTER
+  reapers skipped for DirectPipe connections that have no receive queue
+  (latent bug previously masked by transient task error swallowing).
+- **`send_pump_idle?` made public** on all routing strategies — was
+  accidentally private, crashing `Engine#drain_send_queues` with
+  linger > 0.
+
 ## 0.9.0 — 2026-03-31
 
 ### Breaking

data/README.md

@@ -9,9 +9,9 @@
 
 OMQ builds ZeroMQ socket patterns on top of [protocol-zmtp](https://github.com/paddor/protocol-zmtp) (a pure Ruby [ZMTP 3.1](https://rfc.zeromq.org/spec/23/) codec) using [Async](https://github.com/socketry/async) fibers. It speaks native ZeroMQ on the wire and interoperates with libzmq, pyzmq, CZMQ, and everything else in the ZMQ ecosystem.
 
-> **234k msg/s** inproc | **49k msg/s** ipc | **36k msg/s** tcp
+> **980k msg/s** inproc | **38k msg/s** ipc | **31k msg/s** tcp
 >
-> **12 µs** inproc latency | **51 µs** ipc | **62 µs** tcp
+> **10 µs** inproc latency | **71 µs** ipc | **82 µs** tcp
 >
 > Ruby 4.0 + YJIT on a Linux VM — see [`bench/`](bench/) for full results
 
@@ -30,7 +30,7 @@ See [GETTING_STARTED.md](GETTING_STARTED.md) for a ~30 min walkthrough of all ma
 ## Highlights
 
 - **Zero dependencies on C** — no extensions, no FFI, no libzmq. `gem install` just works everywhere
-- **Fast** — YJIT-optimized hot paths, batched sends, 234k msg/s inproc with 12 µs latency
+- **Fast** — YJIT-optimized hot paths, batched sends, recv prefetching, direct-pipe inproc bypass. 980k msg/s inproc, 10 µs latency
 - **[`omq` CLI](https://github.com/paddor/omq-cli)** — `gem install omq-cli` for a command-line tool with Ruby eval, Ractor parallelism, and script handlers
 - **Every socket pattern** — req/rep, pub/sub, push/pull, dealer/router, xpub/xsub, pair, and all draft types
 - **Every transport** — tcp, ipc (Unix domain sockets), inproc (in-process queues)
@@ -125,6 +125,26 @@ pull.close
 
 The IO thread runs all pumps, reconnection, and heartbeating in the background. When you're inside an `Async` block, OMQ uses the existing reactor instead.
 
+### Queue Interface
+
+All sockets expose an `Async::Queue`-inspired interface:
+
+| Async::Queue | OMQ Socket | Notes |
+|---|---|---|
+| `enqueue(item)` / `push(item)` | `enqueue(msg)` / `push(msg)` | Also: `send(msg)`, `<< msg` |
+| `dequeue(timeout:)` / `pop(timeout:)` | `dequeue(timeout:)` / `pop(timeout:)` | Defaults to socket's `read_timeout` |
+| `wait` | `wait` | Blocks indefinitely (ignores `read_timeout`) |
+| `each` | `each` | Yields messages; returns on close or timeout |
+
+```ruby
+pull = OMQ::PULL.bind('inproc://work')
+
+# iterate messages like a queue
+pull.each do |msg|
+  puts msg.first
+end
+```
+
 ## Socket Types
 
 All sockets are thread-safe. Default HWM is 1000 messages per socket. Classes live under `OMQ::` (alias: `ØMQ`).
@@ -163,6 +183,11 @@ echo "hello" | omq req -c tcp://localhost:5555
 
 See the [omq-cli README](https://github.com/paddor/omq-cli) for full documentation.
 
+## Companion Gems
+
+- **[omq-ffi](https://github.com/paddor/omq-ffi)** — libzmq FFI backend. Same OMQ socket API, but backed by libzmq instead of the pure Ruby ZMTP stack. Useful for interop testing and when you need libzmq-specific features. Requires libzmq installed.
+- **[omq-ractor](https://github.com/paddor/omq-ractor)** — bridge OMQ sockets into Ruby Ractors for true parallel processing across cores. I/O stays on the main Ractor, worker Ractors do pure computation.
+
 ## Development
 
 ```sh

data/lib/omq/channel.rb

@@ -2,12 +2,12 @@
 
 module OMQ
   class CHANNEL < Socket
-    include ZMTP::Readable
-    include ZMTP::Writable
-    include ZMTP::SingleFrame
+    include Readable
+    include Writable
+    include SingleFrame
 
-    def initialize(endpoints = nil, linger: 0)
-      _init_engine(:CHANNEL, linger: linger)
+    def initialize(endpoints = nil, linger: 0, backend: nil)
+      _init_engine(:CHANNEL, linger: linger, backend: backend)
       _attach(endpoints, default: :connect)
     end
   end

data/lib/omq/client_server.rb

@@ -2,23 +2,23 @@
 
 module OMQ
   class CLIENT < Socket
-    include ZMTP::Readable
-    include ZMTP::Writable
-    include ZMTP::SingleFrame
+    include Readable
+    include Writable
+    include SingleFrame
 
-    def initialize(endpoints = nil, linger: 0)
-      _init_engine(:CLIENT, linger: linger)
+    def initialize(endpoints = nil, linger: 0, backend: nil)
+      _init_engine(:CLIENT, linger: linger, backend: backend)
       _attach(endpoints, default: :connect)
     end
   end
 
   class SERVER < Socket
-    include ZMTP::Readable
-    include ZMTP::Writable
-    include ZMTP::SingleFrame
+    include Readable
+    include Writable
+    include SingleFrame
 
-    def initialize(endpoints = nil, linger: 0)
-      _init_engine(:SERVER, linger: linger)
+    def initialize(endpoints = nil, linger: 0, backend: nil)
+      _init_engine(:SERVER, linger: linger, backend: backend)
       _attach(endpoints, default: :bind)
     end