omq-zstd 0.4.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9e725966f3527f9ad15396b9dc6b4a182a93a3cdf5b1f6916b6ccb25ba39b146
+  data.tar.gz: 4c6bc2ed5ba87c77868ff93781dc9269a8929b78df8e190097bc943e84c2c46c
+SHA512:
+  metadata.gz: fa50d84a5d8a53a0a0bad2c5df4cee8d9435f04612f260082d02a226740f438d91f33702e29c3b974fda0ef6302080ab96d61a4e804e0abb409b2892a7f24e92
+  data.tar.gz: 74119b06fa29d1dc339803c153fe2571c9d401d638a6fdc36cf1b5c64e28f2ba951e935ebbe7203475332d58c1e3a4be2622fb2acd5ac5da9ff713122f10d53b

data/DESIGN.md

@@ -0,0 +1,162 @@
+# omq-transport-zstd — Implementation Notes
+
+These are implementation details for the `zstd+tcp://` transport plugin.
+The wire protocol is specified in [RFC.md](RFC.md); this document covers
+the OMQ-specific architecture that doesn't belong in the RFC.
+
+## Why a transport plugin
+
+Compression could live at three layers. Each has a fatal flaw except the
+transport layer.
+
+**Socket-level wrapper** (too high). A wrapper sits above routing and
+knows nothing about transports. It can't distinguish TCP from IPC or
+inproc, so it compresses local connections (pure overhead). It also
+can't act on new connections naturally — dict shipping requires
+per-connection state, but the wrapper only sees messages after routing
+has dispatched them. Reconnect handling requires hooking into
+connection lifecycle events, which is awkward from outside.
+
+**ZMTP connection layer** (too low). Embedding compression into each
+ZMTP connection means PUB fan-out compresses the same message N times
+(once per subscriber connection). The connection layer has no
+socket-wide view, so there's no way to share compression work across
+connections.
+
+**Transport layer** (right). `zstd+tcp://` makes transport selection
+explicit in the endpoint URI. Only TCP connections get compressed. IPC
+and inproc are unaffected even on the same socket. Dict lifetime
+matches connection lifetime naturally (new connection = new wrapper =
+re-ship dict). No negotiation needed — both peers use `zstd+tcp://`.
+The Codec is socket-wide (shared across connections via the Dialer or
+Listener), so PUB compresses once and reuses the result.
+
+## Architecture
+
+```
+Socket         bind("zstd+tcp://...")  /  connect("zstd+tcp://...")
+  │                                         │
+  ▼                                         ▼
+Engine         transport.listener(...)      transport.dialer(...)
+  │                → Listener                 → Dialer
+  │                  holds Codec               holds Codec
+  │                  #wrap_connection           #wrap_connection
+  ▼                        │                        │
+ConnectionLifecycle        ▼                        ▼
+  ready!  ──────►  ZstdConnection(conn, codec)
+                   │ #write_message  → compress → ship_dict! → delegate
+                   │ #receive_message → decode (per-connection recv_dict)
+                   │ #respond_to?(:write_wire) → false (forces fan-out path)
+```
+
+### Dialer / Listener
+
+Both are stateful transport objects created by the transport module's
+`.dialer` / `.listener` factory methods. They hold the Codec and
+implement `#wrap_connection(conn)` which wraps raw ZMTP connections
+in a `ZstdConnection`. The Engine stores them in `@dialers` / `@listeners`
+(Hash keyed by endpoint) and calls `#wrap_connection` during
+`ConnectionLifecycle#ready!`.
+
+Reconnect calls `dialer.connect` directly — no transport lookup or
+opts replay needed. The Dialer holds everything.
+
+### Codec (socket-wide)
+
+One Codec per Dialer or Listener. Shared across all connections of
+that endpoint. Owns:
+
+- **Compression**: `#compress_parts(parts)` with identity cache
+- **Training**: sample collection, `RZstd::Dictionary.train`, dict ID patching
+- **Send dict**: `#send_dict_bytes` — the trained or user-supplied dict bytes
+
+### ZstdConnection (per-connection)
+
+`SimpleDelegator` wrapping a `Protocol::ZMTP::Connection`. Per-connection
+state:
+
+- `@dict_shipped` — whether the dict has been sent on this connection
+- `@recv_dict` — the peer's dictionary for decompression
+
+Intercepts `#send_message`, `#write_message`, `#write_messages`,
+`#receive_message`. Returns `false` for `respond_to?(:write_wire)` to
+force fan-out through `#write_message` (which hits the compression
+cache) instead of pre-encoded wire bytes.
+
+## Identity-based compression cache
+
+PUB fan-out sends the same frozen message parts Array to every
+subscriber's `#write_message`. The Codec exploits this with an
+`Object#equal?` check:
+
+```ruby
+def compress_parts(parts)
+  return @cached_compressed if parts.equal?(@cached_parts)
+  # ... compress ...
+  @cached_parts      = parts
+  @cached_compressed = compressed.freeze
+end
+```
+
+`.equal?` is O(1) — same frozen Array object from `freeze_message`.
+First subscriber pays the compression cost; subsequent subscribers
+get the cached result. Net: one compression per message, N wire
+writes.
+
+## Dict shipping order
+
+Training can trigger DURING `#compress_parts` (when the sample
+threshold is reached mid-compression). The dict must be shipped
+AFTER compression but BEFORE the wire write, so the receiver
+has the dict before seeing frames that use it:
+
+```ruby
+def write_message(parts)
+  compressed = @codec.compress_parts(parts)  # may trigger training
+  ship_dict!                                  # ships if newly trained
+  __getobj__.write_message(compressed)
+end
+```
+
+## Training heuristics
+
+- **Sample threshold**: 1000 messages OR 100 KiB of plaintext, whichever first
+- **Sample size cap**: frames > 1024 bytes are skipped (dictionaries primarily benefit small frames)
+- **Dict capacity**: 8 KiB (conservative; Zstd recommends ~100:1 sample-to-dict ratio)
+- **Dict ID patching**: auto-trained dicts get a random ID in the user range (32768..2^31-1) to avoid collisions with Zstd's built-in dict IDs
+- **Training failure**: if `RZstd::Dictionary.train` raises, training is disabled permanently for the socket. No retry.
+
+## Frame dispatch
+
+Three sentinels for per-part decoding:
+
+| Preamble (4 bytes hex) | Meaning |
+|---|---|
+| `00 00 00 00` | Uncompressed plaintext (part too small or incompressible) |
+| `28 B5 2F FD` | Zstd compressed frame (the standard Zstd magic number) |
+| `37 A4 30 EC` | Zstd dictionary — install into per-connection recv slot |
+
+Dict frames are single-part ZMTP messages. When all parts in a message
+are dict frames, `#decode_parts` returns `nil` and `#receive_message`
+loops to get the next real message.
+
+## Budget enforcement
+
+The receiver tracks a per-message decompressed byte budget derived from
+`max_message_size`. Each part's declared `Frame_Content_Size` is checked
+BEFORE decompression. The budget decreases across parts of a multipart
+message, so the total decompressed size can't exceed the limit even if
+individual parts are within bounds.
+
+## Constants
+
+```
+MAX_DECOMPRESSED_SIZE  = 16 MiB   (absolute cap per frame)
+MAX_DICT_SIZE          = 64 KiB   (reject oversized dicts)
+DICT_CAPACITY          = 8 KiB    (training target size)
+TRAIN_MAX_SAMPLES      = 1000
+TRAIN_MAX_BYTES        = 100 KiB
+TRAIN_MAX_SAMPLE_LEN   = 1024     (skip large frames for training)
+MIN_COMPRESS_NO_DICT   = 512 B
+MIN_COMPRESS_WITH_DICT = 64 B
+```

data/LICENSE

@@ -0,0 +1,15 @@
+ISC License
+
+Copyright (c) 2026, Patrik Wenger
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

data/README.md

@@ -0,0 +1,163 @@
+# omq-zstd
+
+[![Gem Version](https://img.shields.io/gem/v/omq-zstd?color=e9573f)](https://rubygems.org/gems/omq-zstd)
+[![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+
+> **Status:** Draft. Wire format may change before the first tagged release.
+
+Zstandard-compressed TCP transport for [OMQ](https://github.com/paddor/omq).
+Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
+compressed per-part with [Zstandard](https://github.com/facebook/zstd).
+Compression is intrinsic to the transport — no negotiation, no socket option,
+no payload changes. The ZMTP handshake itself runs over plain TCP; only
+post-handshake message parts are compressed.
+
+See [RFC.md](RFC.md) for the wire-format specification and
+[DESIGN.md](DESIGN.md) for the implementation rationale.
+
+## Install
+
+```ruby
+# Gemfile
+gem "omq-zstd"
+```
+
+```sh
+gem install omq-zstd
+```
+
+## Usage
+
+```ruby
+require "omq"
+require "omq/zstd"
+
+pull = OMQ::PULL.new
+push = OMQ::PUSH.new
+
+uri = pull.bind("zstd+tcp://127.0.0.1:0")
+push.connect(uri.to_s)
+
+push << ["hello, compressed world"]
+pull.receive  # => ["hello, compressed world"]
+```
+
+Both peers must use the `zstd+tcp://` scheme. A `tcp://` peer cannot talk to
+a `zstd+tcp://` peer — they speak different transports.
+
+### Compression level
+
+Default is **`-3`** (negative = Zstd's fast strategy). Override at bind/connect:
+
+```ruby
+pull.bind("zstd+tcp://127.0.0.1:0", level: 3)
+push.connect("zstd+tcp://127.0.0.1:5555", level: 9)
+```
+
+Per-direction, per-side: each side picks its own send level. Receiving works
+at any level the peer chose.
+
+### Dictionaries
+
+Small messages don't compress well on their own. A shared Zstd dictionary
+trained on representative payloads gives 2–10× ratios on payloads in the
+dozens-to-hundreds-of-bytes range.
+
+**User-supplied dictionary** (out-of-band agreement):
+
+```ruby
+dict = File.binread("schema.dict")  # produced by `zstd --train`
+push.connect("zstd+tcp://127.0.0.1:5555", dict: dict)
+```
+
+The sender ships the dictionary to the receiver in-band as a one-shot
+single-part message prefixed with the dictionary sentinel
+(`37 A4 30 EC`), so the receiver does not need a copy on disk.
+
+**Auto-trained dictionary** (zero config — the default when no `dict:` is
+passed): the sender collects up to 1000 samples or 100 KiB (whichever hits
+first), trains a dictionary, ships it inline, and switches to dictionary
+mode. Until then, payloads are compressed without a dictionary or sent
+plaintext when below the threshold.
+
+### Compression thresholds
+
+To avoid pessimizing tiny frames, the sender skips compression below:
+
+| Mode | Threshold |
+|------|-----------|
+| No dictionary | 512 B |
+| With dictionary | 64 B |
+
+Below the threshold the part is sent uncompressed (4-byte zero sentinel +
+plaintext bytes).
+
+### Security limits
+
+The receiver bounds decompression by the socket's own `max_message_size`
+— the same knob you'd use on a plain `tcp://` socket. It caps the
+**total decompressed size of all parts in a single message**, not each
+part individually: the budget starts at `max_message_size` and shrinks
+as each part is decoded, so a message whose parts sum to more than the
+cap is rejected on the offending part.
+
+```ruby
+pull.max_message_size = 1_048_576  # 1 MiB cap on the total message
+```
+
+If `max_message_size` is `nil` (OMQ's default, unlimited), there is no
+ceiling on decompressed message size. Set a value that matches what
+your application would tolerate over plain `tcp://`.
+
+Independent of the message-size knob, the dictionary itself is capped at
+64 KiB (Zstd's recommended dictionary size range). A peer attempting to
+ship a larger dictionary, or send a message whose decompressed parts
+exceed `max_message_size`, drops the connection — `OMQ::SocketDeadError`
+surfaces on the next `receive`.
+
+## When to use it
+
+`zstd+tcp://` is worth picking when:
+
+- You're network-bound (cross-region, IoT links, congested LAN).
+- Your payloads have repetitive structure (JSON, log lines, protobuf with
+  string fields, similar binary records).
+- You want compression without touching the message format on either side.
+
+It is **not** worth it for:
+
+- `inproc://` or `ipc://` — irrelevant; there is no wire to shrink. Use
+  `zstd+tcp://` only on the connections that actually need it. Other
+  transports on the same socket are unaffected.
+- Already-compressed payloads (gzip, video, encrypted blobs) — the Zstd
+  pass adds CPU for no gain.
+- Latency-critical sub-microsecond paths — compression adds single-digit
+  microseconds per kilobyte at low levels, but it is not free.
+
+## How it works (in one paragraph)
+
+`require "omq/zstd"` registers the `zstd+tcp` scheme on
+`OMQ::Engine.transports`. A `zstd+tcp` socket builds a per-engine
+`Codec` (one Zstd dictionary instance shared across all the socket's
+connections — fan-out compresses each part exactly once). Each accepted
+or dialed TCP connection is wrapped in `ZstdConnection`, a
+`SimpleDelegator` over the underlying ZMTP connection that intercepts
+`#send_message` / `#write_message` / `#receive_message`. Message parts
+go out as a 4-byte sentinel + payload: `00 00 00 00` for plaintext,
+`28 B5 2F FD` (Zstandard frame magic) for a compressed part, or
+`37 A4 30 EC` for a one-shot single-part dictionary shipment. The
+receiver dispatches on the sentinel, decompresses with bounded
+buffers, and hands plaintext parts up to ZMTP unchanged.
+
+## Development
+
+```sh
+OMQ_DEV=1 bundle install
+OMQ_DEV=1 bundle exec rake test
+OMQ_DEV=1 bundle exec ruby --yjit bench/level_sweep.rb
+```
+
+## License
+
+[ISC](LICENSE)