omq 0.5.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 76cecd3650f9fb73d699a027570421f589452a40cf6bbf207af02afb2bc6df08
-  data.tar.gz: 0252e01bacd5278defe46ed9ab4750b3e33ca71e928beb13f1092c23a7c2ce62
+  metadata.gz: 6cbe3ab5a57c05043fa8f088c106c54ada9268286fe812b4fb43bf0d4ad4bdff
+  data.tar.gz: 7ef9f87c80ee05ea36bc60386cc7221ad4869fadc4840d5dcc58c218bdebfabc
 SHA512:
-  metadata.gz: de2508cbe4e22c6dc93bdeab4626fed0c5280cf74eaaf4c4f886f566719a3c732f5db9b766c201f7e9d6eb7bb00ee4ed2d2392eec232abe88c7847148b889fc5
-  data.tar.gz: 65e96cef70697abe90588e5e35f4563e55ef183bb0be44276c6cb62ec007f6accb4e482a08c14d89b15ec3f542d75c78bacc87c6174ca6f4e69b588ddb540730
+  metadata.gz: d84d69dafd6b1f8e98e22726556f38865e61b9e047e39ea88ab0546dbac224887ae11645728b193b457f99401f6643133dcff357cef03856cf4308818ae05288
+  data.tar.gz: 647d4a79a3cb4ae55611f95bbefbb598dd700244820b5a20158a87c3430ba54434dfe2e7565727ceff37f53bbebc72e7f33f19ac0a7134749dec2a28331d3cf5

data/CHANGELOG.md

@@ -1,5 +1,178 @@
 # Changelog
 
+## 0.6.0 — 2026-03-30
+
+### Added
+
+- **`OMQ::SocketDeadError`** — raised on `#send`/`#receive` after an
+  internal pump task crashes. The original exception is available via
+  `#cause`. The socket is permanently bricked.
+- **`Engine#spawn_pump_task`** — replaces bare `parent_task.async(transient: true)`
+  in all 10 routing strategies. Catches unexpected exceptions and forwards
+  them via `signal_fatal_error` so blocked `#send`/`#receive` callers see
+  the real error instead of deadlocking.
+- **`Socket#close_read`** — pushes a nil sentinel into the recv queue,
+  causing a blocked `#receive` to return nil. Used by `--transient` to
+  drain remaining messages before exit instead of killing the task.
+- **`send_pump_idle?`** on all routing classes — tracks whether the send
+  pump has an in-flight batch. `Engine#drain_send_queues` now waits for
+  both `send_queue.empty?` and `send_pump_idle?`, preventing message loss
+  during linger close.
+- **Grace period after `peer_connected`** — senders that bind or connect
+  to multiple endpoints sleep one `reconnect_interval` (100ms) after the
+  first peer handshake, giving latecomers time to connect before messages
+  start flowing.
+- **`-P/--parallel [N]` for `omq pipe`** — spawns N Ractor workers
+  (default: nproc) in a single process for true CPU parallelism. Each
+  Ractor runs its own Async reactor with independent PULL/PUSH sockets.
+  `$F` in `-e` expressions is transparently rewritten for Ractor isolation.
+- **`BEGIN{}`/`END{}` blocks in `-e` expressions** — like awk, run setup
+  before the message loop and teardown after. Supports nested braces.
+  Example: `-e 'BEGIN{ @sum = 0 } @sum += Integer($_); next END{ puts @sum }'`
+- **`--reconnect-ivl`** — set reconnect interval from the CLI, accepts a
+  fixed value (`0.5`) or a range for exponential backoff (`0.1..2`).
+- **`--transient`** — exit when all peers disconnect (after at least one
+  message has been sent/received). Useful for pipeline sinks and workers.
+- **`--examples`** — annotated usage examples, paged via `$PAGER` or `less`.
+  `--help` now shows help + examples (paged); `-h` shows help only.
+- **`-r` relative paths** — `-r./lib.rb` and `-r../lib.rb` resolve via
+  `File.expand_path` instead of `$LOAD_PATH`.
+- **`peer_connected` / `all_peers_gone`** — `Async::Promise` hooks on
+  `Socket` for connection lifecycle tracking.
+- **`reconnect_enabled=`** — disable auto-reconnect per socket.
+- **Pipeline benchmark** — 4-worker fib pipeline via `omq` CLI
+  (`bench/cli/pipeline.sh`). ~300–1800 msg/s depending on N.
+- **DESIGN.md** — architecture overview covering task trees, send pump
+  batching, ZMTP wire protocol, transports, and the fallacies of
+  distributed computing.
+- **Draft socket types in omqcat** — CLIENT, SERVER, RADIO, DISH, SCATTER,
+  GATHER, CHANNEL, and PEER are now supported in the CLI tool.
+  - `-j`/`--join GROUP` for DISH (like `--subscribe` for SUB)
+  - `-g`/`--group GROUP` for RADIO publishing
+  - `--target` extended to SERVER and PEER (accepts `0x` hex for binary routing IDs)
+  - `--echo` and `-e` on SERVER/PEER reply to the originating client via `send_to`
+  - CLIENT uses request-reply loop (send then receive)
+- **Unified `--timeout`** — replaces `--recv-timeout`/`--send-timeout` with a
+  single `-t`/`--timeout` flag that applies to both directions.
+- **`--linger`** — configurable drain time on close (default 5s).
+- **Exit codes** — 0 = success, 1 = error, 2 = timeout.
+- **CLI unit tests** — 74 tests covering Formatter, routing helpers,
+  validation, and option parsing.
+- **Quantized `--interval`** — uses `Async::Loop.quantized` for
+  wall-clock-aligned, start-to-start timing (no drift).
+- **`-e` as data source** — eval expressions can generate messages without
+  `--data`, `--file`, or stdin. E.g. `omq pub -e 'Time.now.to_s' -i 1`.
+- **`$_` in eval** — set to the first frame of `$F` inside `-e` expressions,
+  following Ruby convention.
+- **`wait_for_peer`** — connecting sockets wait for the first peer handshake
+  before sending. Replaces the need for manual `--delay` on PUB, PUSH, etc.
+- **`OMQ_DEV` env var** — unified dev-mode flag for loading local omq and
+  omq-curve source via `require_relative` (replaces `DEV_ENV`).
+- **`--marshal` / `-M`** — Ruby Marshal stream format. Sends any Ruby
+  object over the wire; receiver deserializes and prints `inspect` output.
+  E.g. `omq pub -e 'Time.now' -M` / `omq sub -M`.
+- **`-e` single-shot** — eval runs once and exits when no other data
+  source is present. Supports `self << msg` for direct socket sends.
+- **`subscriber_joined`** — `Async::Promise` on PUB/XPUB that resolves
+  when the first subscription arrives. CLI PUB waits for it before sending.
+- **`#to_str` enforcement** — message parts must be string-like; passing
+  integers or symbols raises `NoMethodError` instead of silently coercing.
+- **`-e` error handling** — eval errors abort with exit code 3.
+- **`--raw` outputs ZMTP frames** — flags + length + body per frame,
+  suitable for `hexdump -C`. Compression remains transparent.
+- **ROUTER `router_mandatory` by default** — CLI ROUTER rejects sends to
+  unknown identities and waits for first peer before sending.
+- **`--timeout` applies to `wait_for_peer`** — `-t` now bounds the initial
+  connection wait via `Async::TimeoutError`.
+
+### Improved
+
+- **Received messages are always frozen** — `Connection#receive_message`
+  (TCP/IPC) now returns a frozen array of frozen strings, matching the
+  inproc fast-path. REP and REQ recv transforms rewritten to avoid
+  in-place mutation (`Array#shift` → slicing).
+- **CLI refactored into 16 files** — the 1162-line `cli.rb` monolith is
+  decomposed into `CLI::Config` (frozen `Data.define`), `CLI::Formatter`,
+  `CLI::BaseRunner` (shared infrastructure), and one runner class per
+  socket type combo (PushRunner, PullRunner, ReqRunner, RepRunner, etc.).
+  Each runner models its behavior as a single `#run_loop` override.
+- **`--transient` uses `close_read` instead of `task.stop`** — recv-only
+  and bidirectional sockets drain their recv queue via nil sentinel before
+  exiting, preventing message loss on disconnect. Send-only sockets still
+  use `task.stop`.
+- **Pipeline benchmark** — natural startup order (producer → workers →
+  sink), workers use `--transient -t 1` (timeout covers workers that
+  connect after the producer is already gone). Verified correct at 5M messages
+  (56k msg/s sustained, zero message loss).
+- **Renamed `omqcat` → `omq`** — the CLI executable is now `omq`, matching
+  the gem name.
+- **Per-connection task subtrees** — each connection gets an isolated Async
+  task whose children (heartbeat, recv pump, reaper) are cleaned up
+  automatically when the connection dies. No reparenting.
+- **Flat task tree** — send pump spawned at socket level (singleton), not
+  inside connection subtrees. Accept loops use `defer_stop` to prevent
+  socket leaks on stop.
+- **`compile_expr`** — `-e` expressions compiled once as a proc,
+  `instance_exec` per message (was `instance_eval` per message).
+- **Close lifecycle** — stop listeners before drain only when connections
+  exist; keep listeners open with zero connections so late-arriving peers
+  can receive queued messages during linger.
+- **Reconnect guard** — `@closing` flag suppresses reconnect during close.
+- **Task annotations** — all pump tasks carry descriptive annotations
+  (send pump, recv pump, reaper, heartbeat, reconnect, tcp/ipc accept).
+- **Rename monitor → reaper** — clearer name for PUSH/SCATTER dead-peer
+  detection tasks.
+- **Extracted `OMQ::CLI` module** — `exe/omq` is a thin wrapper;
+  bulk of the CLI lives in `lib/omq/cli.rb` (loaded via `require "omq/cli"`,
+  not auto-loaded by `require "omq"`).
+  - `Formatter` class for encode/decode/compress/decompress
+  - `Runner` is stateful with `@sock`, cleaner method signatures
+- **Quoted format uses `String#dump`/`undump`** — fixes backslash escaping
+  bug, proper round-tripping of all byte values.
+- **Hex routing IDs** — binary identities display as `0xdeadbeef` instead
+  of lossy Z85 encoding. `--target 0x...` decodes hex on input.
+- **Compression-safe routing** — routing ID and delimiter frames are no
+  longer compressed/decompressed in ROUTER, SERVER, and PEER loops.
+- **`require_relative` in CLI** — `exe/omq` loads the local source tree
+  instead of the installed gem.
+- **`output` skips nil** — `-e` returning nil no longer prints a blank line.
+- **Removed `#count_reached?`** — inlined for clarity.
+- **System tests overhauled** — `test/omqcat` → `test/cli`, all IPC
+  abstract namespace, `set -eu`, stderr captured, no sleeps (except
+  ROUTER --target), under 10s.
+
+### Fixed
+
+- **Inproc DEALER→REP broker deadlock** — `Writable#send` freezes the
+  message array, but the REP recv transform mutated it in-place via
+  `Array#shift`. On the inproc fast-path the frozen array passed through
+  the DEALER send pump unchanged, causing `FrozenError` that silently
+  killed the send pump task and deadlocked the broker.
+- **Pump errors swallowed silently** — all send/recv pump tasks ran as
+  `transient: true` Async tasks, so unexpected exceptions (bugs) were
+  logged but never surfaced to the caller. The socket would deadlock
+  instead of raising. Now `Engine#signal_fatal_error` stores the error
+  and unblocks the recv queue; subsequent `#send`/`#receive` calls
+  re-raise it as `SocketDeadError`. Expected errors (`Async::Stop`,
+  `ProtocolError`, `CONNECTION_LOST`) are still handled normally.
+- **Pipe `--transient` drains too early** — `all_peers_gone` fired while
+  `pull.receive` was blocked, hanging the worker forever. Now the transient
+  monitor pushes a nil sentinel via `close_read`, which unblocks the
+  blocked dequeue and lets the loop drain naturally.
+- **Linger drain missed in-flight batches** — `drain_send_queues` only
+  checked `send_queue.empty?`, but the send pump may have already dequeued
+  messages into a local batch. Now also checks `send_pump_idle?`.
+- **Socket option delegators not Ractor-safe** — `define_method` with a
+  block captured state from the main Ractor, causing `Ractor::IsolationError`
+  when calling setters like `recv_timeout=`. Replaced with `Forwardable`.
+- **Pipe endpoint ordering** — `omq pipe -b url1 -c url2` assigned PULL
+  to `url2` and PUSH to `url1` (backwards) because connects were
+  concatenated before binds. Now uses ordered `Config#endpoints`.
+- **Linger drain kills reconnect tasks** — `Engine#close` set `@closed = true`
+  before draining send queues, causing reconnect tasks to bail immediately.
+  Messages queued before any peer connected were silently dropped. Now `@closed`
+  is set after draining, so reconnection continues during the linger period.
+
 ## 0.5.1 — 2026-03-28
 
 ### Improved

data/README.md

@@ -33,6 +33,8 @@ Modern Ruby has closed the gap:
 
 When [CZTop](https://github.com/paddor/cztop) was written, none of this existed. Today, a pure Ruby ZMTP implementation is fast enough for production use — and you get `gem install` with no compiler toolchain, no system packages, and no segfaults.
 
+See [DESIGN.md](DESIGN.md) for architecture details: task trees, send pump batching, the ZMTP wire protocol, and how OMQ handles the fallacies of distributed computing.
+
 ## Install
 
 No system libraries needed — just Ruby:
@@ -142,56 +144,56 @@ Benchmarked with benchmark-ips on Linux x86_64 (Ruby 4.0.2 +YJIT):
 
 See [`bench/`](bench/) for full results and scripts.
 
-## omqcat — CLI tool
+## omq — CLI tool
 
-`omqcat` is a command-line tool for sending and receiving messages on any OMQ socket. Like `nngcat` from libnng, but with Ruby superpowers.
+`omq` is a command-line tool for sending and receiving messages on any OMQ socket. Like `nngcat` from libnng, but with Ruby superpowers.
 
 ```sh
 # Echo server
-omqcat rep -b tcp://:5555 --echo
+omq rep -b tcp://:5555 --echo
 
 # Upcase server in one line
-omqcat rep -b tcp://:5555 -e '$F.map(&:upcase)'
+omq rep -b tcp://:5555 -e '$F.map(&:upcase)'
 
 # Client
-echo "hello" | omqcat req -c tcp://localhost:5555
+echo "hello" | omq req -c tcp://localhost:5555
 # => HELLO
 
 # PUB/SUB
-omqcat sub -b tcp://:5556 -s "weather."  &
-echo "weather.nyc 72F" | omqcat pub -c tcp://localhost:5556 -d 0.3
+omq sub -b tcp://:5556 -s "weather."  &
+echo "weather.nyc 72F" | omq pub -c tcp://localhost:5556 -d 0.3
 
 # Pipeline with filtering
-tail -f /var/log/syslog | omqcat push -c tcp://collector:5557
-omqcat pull -b tcp://:5557 -e '$F.first.include?("error") ? $F : nil'
+tail -f /var/log/syslog | omq push -c tcp://collector:5557
+omq pull -b tcp://:5557 -e '$F.first.include?("error") ? $F : nil'
 
 # Multipart messages via tabs
-printf "routing-key\tpayload data" | omqcat push -c tcp://localhost:5557
-omqcat pull -b tcp://:5557
+printf "routing-key\tpayload data" | omq push -c tcp://localhost:5557
+omq pull -b tcp://:5557
 # => routing-key	payload data
 
 # JSONL for structured data
-echo '["key","value"]' | omqcat push -c tcp://localhost:5557 -J
-omqcat pull -b tcp://:5557 -J
+echo '["key","value"]' | omq push -c tcp://localhost:5557 -J
+omq pull -b tcp://:5557 -J
 
 # Zstandard compression
-omqcat push -c tcp://remote:5557 -z < data.txt
-omqcat pull -b tcp://:5557 -z
+omq push -c tcp://remote:5557 -z < data.txt
+omq pull -b tcp://:5557 -z
 
 # CURVE encryption
-omqcat rep -b tcp://:5555 -D "secret" --curve-server
+omq rep -b tcp://:5555 -D "secret" --curve-server
 # prints: OMQ_SERVER_KEY='...'
-omqcat req -c tcp://localhost:5555 --curve-server-key '...'
+omq req -c tcp://localhost:5555 --curve-server-key '...'
 ```
 
 The `-e` flag runs Ruby inside the socket instance — the full socket API (`self <<`, `send`, `subscribe`, ...) is available. Use `-r` to require gems:
 
 ```sh
-omqcat sub -c tcp://localhost:5556 -s "" -r json \
+omq sub -c tcp://localhost:5556 -s "" -r json \
   -e 'JSON.parse($F.first)["temperature"]'
 ```
 
-Formats: `--ascii` (default, tab-separated), `--quoted`, `--raw`, `--jsonl`, `--msgpack`. See `omqcat --help` for all options.
+Formats: `--ascii` (default, tab-separated), `--quoted`, `--raw`, `--jsonl`, `--msgpack`. See `omq --help` for all options.
 
 ## Interop with native ZMQ
 

data/exe/omq

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+Warning[:experimental] = false
+
+require_relative "../lib/omq/cli"
+OMQ::CLI.run