omq-zstd 0.4.4 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 22b1f25991e512b445fe5a7202d2671048159b494a7ac5418d1b121f696940ab
-  data.tar.gz: 941b1c5f85ec8dd1a9816fbfac241f882be1f2e9cd80696b1cf894465e90d884
+  metadata.gz: 0adb0f7516e8bfc1df21c5406669d36834d933851dbadb6b41c82d0b961509f7
+  data.tar.gz: 112a512e7266d6761e69a0b87bc2b4656afefc63f0b878c0c120e337206c020e
 SHA512:
-  metadata.gz: 6525d971daf7707cb46113848b2a70c6e29775f9d7a87c6e9dcbd13cd481eebdd7c6a75c919d629b281e491c3263ec56b2e149bc573e4e0f24df32f042247dab
-  data.tar.gz: 99d1a5e3d2ce007036a0712fc7912331bbda35437eca0976d55eb58b8d0eedb52fd5a285c9c025fa658c75231ff4e538fb0f1dcd71d0d38a6d67505a72c4ae1e
+  metadata.gz: b9ef977a6b1a8a1b08935340bd2efb201aacc6adc20dc51f652a68e238c24d459d8afb9f3294699d931721f16b94afa053267a98dc69525f977abec0d4f9e932
+  data.tar.gz: 14e63f2bcbb48f191b13ef2ca3d27b503cd7650e9c1c6b82986c907cdbcfa7de9c6f3e9a644633d8d74ee99513aa147ef1c1545d1904aa964439e05bb4a7cfb3

data/DESIGN.md

@@ -1,161 +1,48 @@
-# omq-transport-zstd — Implementation Notes
+# Zstd transport architecture
 
-These are implementation details for the `zstd+tcp://` transport plugin.
-The wire protocol is specified in [RFC.md](RFC.md); this document covers
-the OMQ-specific architecture that doesn't belong in the RFC.
+The wire contract is [ZMTP-Zstd revision 2](RFC.md). Compression starts
+only after the normal ZMTP handshake. Commands remain uncompressed.
 
-## Why a transport plugin
+## Send path
 
-Compression could live at three layers. Each has a fatal flaw except the
-transport layer.
+Routing applies subscription filters and queue limits to logical messages.
+The transport receives the messages admitted for one connection and encodes
+isolated or large messages per part. Already queued small messages may share
+an arena, targeting 32 KiB without waiting for additional traffic.
 
-**Socket-level wrapper** (too high). A wrapper sits above routing and
-knows nothing about transports. It can't distinguish TCP from IPC or
-inproc, so it compresses local connections (pure overhead). It also
-can't act on new connections naturally — dict shipping requires
-per-connection state, but the wrapper only sees messages after routing
-has dispatched them. Reconnect handling requires hooking into
-connection lifecycle events, which is awkward from outside.
+A codec owns the send dictionary, training state, and compression settings.
+It is shared by the endpoints of one socket. Individual-message encoding
+can reuse the result for the same frozen message across subscribers.
+Arenas are built from each subscriber's admitted sequence.
 
-**ZMTP connection layer** (too low). Embedding compression into each
-ZMTP connection means PUB fan-out compresses the same message N times
-(once per subscriber connection). The connection layer has no
-socket-wide view, so there's no way to share compression work across
-connections.
+Dictionary shipment is tracked per connection and direction. A newly
+connected peer receives the dictionary before any data that uses it.
+Training is optional, uses a 2 KiB dictionary, and collects up to 1000 samples
+or 100 KiB. Samples of 2048 bytes or more are excluded.
 
-**Transport layer** (right). `zstd+tcp://` makes transport selection
-explicit in the endpoint URI. Only TCP connections get compressed. IPC
-and inproc are unaffected even on the same socket. Dict lifetime
-matches connection lifetime naturally (new connection = new wrapper =
-re-ship dict). No negotiation needed — both peers use `zstd+tcp://`.
-The Codec is socket-wide (shared across connections via the Dialer or
-Listener), so PUB compresses once and reuses the result.
+## Receive path
 
-## Architecture
+Each connection owns its receive dictionary and any unread arena messages.
+It accepts raw parts, Zstandard frames, one dictionary shipment, and raw or
+compressed `ZSTA` arenas. Dictionary shipments must be single-part messages.
 
-```
-Socket         bind("zstd+tcp://...")  /  connect("zstd+tcp://...")
-  │                                         │
-  ▼                                         ▼
-Engine         transport.listener(...)      transport.dialer(...)
-  │                → Listener                 → Dialer
-  │                  holds Codec               holds Codec
-  │                  #wrap_connection           #wrap_connection
-  ▼                        │                        │
-ConnectionLifecycle        ▼                        ▼
-  ready!  ──────►  ZstdConnection(conn, codec)
-                   │ #write_message  → compress → ship_dict! → delegate
-                   │ #receive_message → decode (per-connection recv_dict)
-                   │ #respond_to?(:write_wire) → false (forces fan-out path)
-```
+The shared compression transport layer validates an entire arena before
+returning its first message. Each subsequent receive returns one logical
+message, preserving queue backpressure and receive scheduling. An invalid
+arena closes its connection without exposing a valid prefix or killing the
+socket's other connections.
 
-### Dialer / Listener
+## Bounds
 
-Both are stateful transport objects created by the transport module's
-`.dialer` / `.listener` factory methods. They hold the Codec and
-implement `#wrap_connection(conn)` which wraps raw ZMTP connections
-in a `ZstdConnection`. The Engine stores them in `@dialers` / `@listeners`
-(Hash keyed by endpoint) and calls `#wrap_connection` during
-`ConnectionLifecycle#ready!`.
+- Arena envelope: at most 65552 bytes.
+- Decoded arena: at most 65536 bytes, 256 messages, and 4096 parts.
+- Dictionary: at most 8192 bytes, shipped once per direction.
+- Application size limit: checked per logical message inside an arena,
+  including a 64-byte storage charge per part.
 
-Reconnect calls `dialer.connect` directly — no transport lookup or
-opts replay needed. The Dialer holds everything.
+Zstandard frames must declare their content size and consume exactly their
+wire part. Arena frames must declare the same size as their arena header.
+Decompression is bounded by that size. The outer ZMTP framer allows transport
+envelopes independently of the application message limit.
 
-### Codec (socket-wide)
-
-One Codec per Dialer or Listener. Shared across all connections of
-that endpoint. Owns:
-
-- **Compression**: `#compress_parts(parts)` with identity cache
-- **Training**: sample collection, `Zrip::DictTrainer`, dict ID patching
-- **Send dict**: `#send_dict_bytes` — the trained or user-supplied dict bytes
-
-### ZstdConnection (per-connection)
-
-`SimpleDelegator` wrapping a `Protocol::ZMTP::Connection`. Per-connection
-state:
-
-- `@dict_shipped` — whether the dict has been sent on this connection
-- `@recv_dict` — the peer's dictionary for decompression
-
-Intercepts `#send_message`, `#write_message`, `#write_messages`,
-`#receive_message`. Returns `false` for `respond_to?(:write_wire)` to
-force fan-out through `#write_message` (which hits the compression
-cache) instead of pre-encoded wire bytes.
-
-## Identity-based compression cache
-
-PUB fan-out sends the same frozen message parts Array to every
-subscriber's `#write_message`. The Codec exploits this with an
-`Object#equal?` check:
-
-```ruby
-def compress_parts(parts)
-  return @cached_compressed if parts.equal?(@cached_parts)
-  # ... compress ...
-  @cached_parts      = parts
-  @cached_compressed = compressed.freeze
-end
-```
-
-`.equal?` is O(1) — same frozen Array object from `freeze_message`.
-First subscriber pays the compression cost; subsequent subscribers
-get the cached result. Net: one compression per message, N wire
-writes.
-
-## Dict shipping order
-
-Training can trigger DURING `#compress_parts` (when the sample
-threshold is reached mid-compression). The dict must be shipped
-AFTER compression but BEFORE the wire write, so the receiver
-has the dict before seeing frames that use it:
-
-```ruby
-def write_message(parts)
-  compressed = @codec.compress_parts(parts)  # may trigger training
-  ship_dict!                                  # ships if newly trained
-  __getobj__.write_message(compressed)
-end
-```
-
-## Training heuristics
-
-- **Sample threshold**: 1000 messages OR 100 KiB of plaintext, whichever first
-- **Sample size cap**: frames > 1024 bytes are skipped (dictionaries primarily benefit small frames)
-- **Dict capacity**: 8 KiB (conservative; Zstd recommends ~100:1 sample-to-dict ratio)
-- **Dict ID patching**: auto-trained dicts get a random ID in the user range (32768..2^31-1) to avoid collisions with Zstd's built-in dict IDs
-- **Training failure**: if `Zrip::DictTrainer#train` raises, training is disabled permanently for the socket. No retry.
-
-## Frame dispatch
-
-Three sentinels for per-part decoding:
-
-| Preamble (4 bytes hex) | Meaning |
-|---|---|
-| `00 00 00 00` | Uncompressed plaintext (part too small or incompressible) |
-| `28 B5 2F FD` | Zstd compressed frame (the standard Zstd magic number) |
-| `37 A4 30 EC` | Zstd dictionary — install into per-connection recv slot |
-
-Dict frames are single-part ZMTP messages. When all parts in a message
-are dict frames, `#decode_parts` returns `nil` and `#receive_message`
-loops to get the next real message.
-
-## Budget enforcement
-
-The receiver tracks a per-message decompressed byte budget derived from
-`max_message_size`. Each part's declared `Frame_Content_Size` is checked
-BEFORE decompression. The budget decreases across parts of a multipart
-message, so the total decompressed size can't exceed the limit even if
-individual parts are within bounds.
-
-## Constants
-
-```
-MAX_DICT_SIZE          = 64 KiB   (reject oversized dicts)
-DICT_CAPACITY          = 8 KiB    (training target size)
-TRAIN_MAX_SAMPLES      = 1000
-TRAIN_MAX_BYTES        = 100 KiB
-TRAIN_MAX_SAMPLE_LEN   = 1024     (skip large frames for training)
-MIN_COMPRESS_NO_DICT   = 512 B
-MIN_COMPRESS_WITH_DICT = 64 B
-```
+Encrypted mechanisms are rejected on compressed endpoints.

data/README.md

@@ -11,8 +11,8 @@ Use [`omq-lz4`](../omq-lz4) for CPU- or memory-sensitive compressed TCP
 work. This gem remains for research, comparison, OMQ.rs parity, and cases
 where zstd ratio matters more than transport simplicity.
 
-Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
-compressed per-part with [Zstandard](https://github.com/facebook/zstd).
+Pick `zstd+tcp://` instead of `tcp://` for individual parts or batches
+compressed with [Zstandard](https://github.com/facebook/zstd).
 Compression is intrinsic to the transport: no negotiation, no socket option,
 no payload changes. The ZMTP handshake runs over plain TCP. Only
 post-handshake message parts are compressed.
@@ -20,6 +20,19 @@ post-handshake message parts are compressed.
 See [RFC.md](RFC.md) for the wire-format specification and
 [DESIGN.md](DESIGN.md) for the implementation rationale.
 
+Revision 2 `ZSTA` arenas are supported, including raw and compressed modes
+and dictionaries. Each arena is validated completely before delivery and
+is limited to 64 KiB, 256 messages, and 4096 parts. `max_message_size`
+applies separately to each decoded message, charging payload bytes plus
+64 bytes per part for storage. Message order and receive HWM apply to the
+individual messages inside an arena.
+
+The sender batches already queued small messages, targeting 32 KiB per arena.
+It never waits to fill a batch. Subscription filtering and queue admission
+happen before batching. Isolated or larger messages use per-part encoding.
+Both peers must support revision 2; no revision negotiation is performed.
+Encrypted mechanisms such as CURVE cannot be combined with this transport.
+
 ## Install
 
 ```ruby

data/RFC.md

@@ -3,6 +3,7 @@
 | Field    | Value                                              |
 |----------|----------------------------------------------------|
 | Status   | Draft                                              |
+| Revision | 2                                                  |
 | Editor   | Patrik Wenger                                      |
 | Scheme   | `zstd+tcp://`                                      |
 | Requires | [RFC 37/ZMTP 3.1](https://rfc.zeromq.org/spec/37/) |
@@ -11,13 +12,14 @@
 ## 1. Abstract
 
 This specification defines `zstd+tcp://`, a TCP transport for ZMTP 3.1
-that applies per-part Zstandard compression after the ZMTP handshake.
+that applies Zstandard compression to parts or bounded arenas of complete
+messages after the ZMTP handshake.
 Both peers use the `zstd+tcp://` scheme in their endpoint URIs. The ZMTP
 greeting and handshake proceed over raw TCP exactly as they would over
 `tcp://`. After the handshake completes, every message part on the wire
-is individually encoded with a 4-byte sentinel dispatch that
-distinguishes uncompressed plaintext, Zstandard-compressed frames, and
-dictionary shipments. No ZMTP properties, command frames, or
+uses a 4-byte sentinel dispatch that
+distinguishes uncompressed plaintext, Zstandard-compressed frames,
+dictionary shipments, and arenas. No ZMTP properties, command frames, or
 negotiation are involved. Compression is an intrinsic property of the
 transport, like encryption is an intrinsic property of TLS.
 
@@ -102,10 +104,10 @@ per-subscriber CPU dominates the total budget.
 ### 4.1 Goals
 
 - Transparent to application code: send/receive operations see plaintext.
-- Per-part sender decision: opt out for short or incompressible parts.
+- Per-part or per-arena sender decision: opt out for short or incompressible data.
 - Works for legacy multipart socket types (PUSH/PULL, PUB/SUB, ...) and
   draft single-frame types alike.
-- Small-message-friendly via an optional shared dictionary, either
+- Small-message-friendly via arenas and an optional shared dictionary, either
   supplied out of band or automatically trained from early traffic.
 - No ZMTP-level negotiation, no new READY properties, no new command
   frames.
@@ -119,8 +121,8 @@ per-subscriber CPU dominates the total budget.
   compression is pure overhead; `ipc://` rarely benefits).
 - Replacing or weakening CurveZMQ or any other security mechanism.
   See Sec. 9.
-- Streaming / context-takeover compression. Each part is decodable in
-  isolation with no dependency on a previous part's LZ77 history.
+- Streaming / context-takeover compression. Parts and arenas have no
+  dependency on earlier compression history beyond the installed dictionary.
 
 
 ## 5. Terminology
@@ -133,14 +135,14 @@ per-subscriber CPU dominates the total budget.
 | Compressed part     | A wire part whose first 4 bytes are the Zstandard magic `28 B5 2F FD`. |
 | Dictionary part     | A wire part whose first 4 bytes are `37 A4 30 EC` (Sec. 7).         |
 | Dictionary message  | A single-part ZMTP message consisting of exactly one dictionary part. |
+| Arena               | A bounded single-part carrier containing complete logical messages. |
 
 
 ## 6. Part Encoding
 
-After the ZMTP handshake completes, every message part on the wire is
-individually encoded. The ZMTP MORE flag is carried on the wire frame
-header as normal. Multipart messages are encoded part by part; each
-part is independent.
+After the ZMTP handshake, data uses per-part encoding or an arena (Sec. 6.7).
+Per-part encoding carries the ZMTP MORE flag normally; each part is independent.
+An arena carries its logical multipart boundaries inside one outer frame.
 
 ### 6.1 Sentinel dispatch
 
@@ -151,16 +153,18 @@ The first 4 bytes of each wire part determine how it is decoded.
 | `00 00 00 00`    | Uncompressed plaintext (Sec. 6.3)    |
 | `28 B5 2F FD`    | Zstandard compressed frame (Sec. 6.4)|
 | `37 A4 30 EC`    | Dictionary shipment (Sec. 7)         |
+| `5A 53 54 41`    | `ZSTA` arena (Sec. 6.7)              |
 
 All other 4-byte values are reserved. A receiver that encounters an
 unknown sentinel MUST close the connection.
 
 ### 6.2 Compression level
 
-The default compression level is **-3** (Zstandard fast strategy). At
-this level the encoder cost is in the low single-digit microseconds per
-kilobyte, and the achieved ratio is within a few percent of level 3 once
-a dictionary is in play.
+This specification suggests **level 1** as an implementation default.
+This keeps encode throughput high while improving wire ratio over
+negative fast levels. This is not a wire requirement. Implementations
+may choose any default level that fits their deployment goals, and
+bandwidth-constrained deployments may choose higher levels such as 3.
 
 The compression level is a sender choice and is not communicated on the
 wire. The receiver decodes any valid Zstandard frame regardless of the
@@ -252,13 +256,73 @@ For each incoming wire part, the receiver proceeds as follows:
 
 4. Sentinel `37 A4 30 EC`: dictionary shipment. See Sec. 7.
 
-5. Any other sentinel: close the connection.
+5. Sentinel `5A 53 54 41`: arena. See Sec. 6.7.
 
-The maximum message size always refers to the **decompressed** plaintext
+6. Any other sentinel: close the connection.
+
+The outer framer MUST allow the bounded arena envelope independently of
+the logical message limit. The maximum message size refers to **decompressed** plaintext
 summed across all parts of a multipart message. A multipart message
-whose total wire length is small but whose total decompressed size
-exceeds the limit MUST be rejected before decoder invocation.
+in per-part encoding whose total decompressed size exceeds the limit MUST
+be rejected before decoder invocation. Arenas use the separate bounds below.
+
+
+### 6.7 Arena (`ZSTA`)
+
+An arena is one single-part, non-command outer ZMTP message:
+
+```
+ZSTA | mode u8 | reserved[3] | decoded_size u32 LE | messages u32 LE | data
+```
 
+The header is 16 bytes. `mode = 0` carries raw arena bytes; `mode = 1`
+carries one Zstandard frame, using the installed dictionary if present.
+The frame MUST carry `Frame_Content_Size` equal to `decoded_size`.
+Reserved bytes MUST be zero; other modes MUST close the connection.
+
+Decoded data concatenates ordinary ZMTP data frames: flags, short or long
+length, then payload. MORE chains parts of one logical message; a clear
+MORE ends that message. Commands, reserved flag bits, incomplete headers,
+truncated bodies, unfinished multipart messages, and trailing bytes are
+invalid. Payloads are application bytes, with no inner compression sentinels.
+
+Protocol limits, checked before decompression:
+
+- `1 <= decoded_size <= 65536` bytes, including inner frame headers.
+- `1 <= messages <= 256`.
+- The outer payload MUST NOT exceed 65552 bytes, including its header.
+- Raw data length MUST equal `decoded_size`; compressed data MUST decode
+  to exactly that length, with no unused compressed input.
+
+The receiver MUST validate the complete decoded arena before exposing any
+message. It MUST contain exactly `messages` complete messages and at most
+4096 parts total. Apply `max_message_size` separately to each logical
+message, including the implementation's per-part storage charge; the arena
+size is a separate bound, never one application's message budget.
+
+Receivers MAY retain one decoded arena and deliver it incrementally.
+Delivery MUST preserve message order, message-level HWM/rate accounting,
+and fair scheduling. Backpressure MUST retain the unread suffix. An arena
+is transport overhead and MUST NOT appear as an application message.
+
+### 6.8 Arena sender rules
+
+Senders MAY batch already queued small messages. They MUST bound both
+bytes and message count and MUST NOT wait for additional messages to fill
+an arena. The recommended target is 32 KiB of decoded frame bytes;
+isolated or larger messages MAY use the existing per-part encodings.
+
+Apply the compression threshold to the whole arena. Use mode 0 below
+threshold or when the compressed block is no smaller than the raw arena.
+Each arena is independent of earlier arenas. A dictionary shipment MUST
+precede any arena that uses it; dictionary changes cannot occur inside
+an arena. Training MAY sample logical messages before encoding the arena.
+
+Fan-out senders MUST filter and admit logical messages before sharing an
+arena. Recipients sharing compressed bytes MUST receive the same message
+sequence and dictionary. Limited credit or message-level eviction MAY
+require smaller arenas or per-message encoding; batching MUST NOT change
+the socket's drop policy or allow a slow peer to starve ready peers.
 
 ## 7. Dictionary Shipment
 
@@ -301,7 +365,7 @@ dictionary-load operation.
   per connection. A receiver that receives a second dictionary message
   on the same connection MUST close the connection.
 
-- A dictionary message MUST be sent BEFORE any compressed part that
+- A dictionary message MUST be sent BEFORE any compressed part or arena that
   references the dictionary. In practice this means the sender ships
   the dictionary before (or immediately after training triggers during)
   the first compressed write that would benefit from it.
@@ -367,8 +431,9 @@ support auto-training:
 | Max sample length      | 2048 bytes                               |
 | Training algorithm     | FastCOVER                                |
 
-Auto-training SHOULD be disabled by default. Applications MAY enable it
-explicitly or supply an out-of-band dictionary instead.
+Whether auto-training is enabled by default is an implementation
+choice. Applications MAY disable it or supply an out-of-band
+dictionary instead.
 
 ### 7.6 Dictionary ID
 
@@ -402,8 +467,8 @@ negotiation in the READY handshake is unaffected.
 
 ### 8.4 Peer requirement
 
-Both peers of a connection MUST use `zstd+tcp://`. There is no
-fallback to plaintext TCP and no negotiation. A `zstd+tcp://` peer
+Both peers MUST use `zstd+tcp://` and support revision 2 arenas. There is no
+fallback to older revisions or plaintext TCP, and no negotiation. A `zstd+tcp://` peer
 connecting to a plain `tcp://` peer (or vice versa) will see garbled
 data or sentinel errors and the connection will fail.
 
@@ -425,10 +490,11 @@ opt-in.
 
 ### 9.2 Length side-channel
 
-Compression makes the wire length of a part depend on its content. An
+Compression makes the wire length of a part or arena depend on its content. An
 on-path observer can learn something about the plaintext from the
 compressed length alone. Deployments that care about traffic analysis
-MUST NOT rely on `zstd+tcp://` to hide payload shape.
+MUST NOT rely on `zstd+tcp://` to hide payload shape. Arena messages share
+compression history; separate logical messages are not separate compression contexts.
 
 ### 9.3 Dictionary contents
 
@@ -443,9 +509,10 @@ received dictionaries across connections.
 A small compressed frame can decompress to many megabytes of plaintext.
 The receiver rules in Sec. 6.6 mitigate this:
 
-1. Every compressed part MUST carry `Frame_Content_Size`. The receiver
+1. Every compressed part or arena MUST carry `Frame_Content_Size`. The receiver
    checks the declared total against the maximum message size before
-   invoking the decoder, so a bomb is rejected on its header alone.
+   invoking the decoder for per-part encoding. Arenas instead enforce the
+   64 KiB decoded bound first, then validate every logical message before delivery.
 2. The decoder is invoked in bounded mode. It aborts if it would write
    more bytes than declared. A peer that lies in the header cannot
    expand a part past its declared size.
@@ -462,7 +529,11 @@ unbounded.
 | Uncompressed sentinel   | `00 00 00 00`                                 |
 | Zstd frame sentinel     | `28 B5 2F FD` (Zstandard frame magic)         |
 | Dictionary sentinel     | `37 A4 30 EC`                                 |
-| Default level           | -3                                            |
+| Arena sentinel          | `5A 53 54 41` (`ZSTA`)                       |
+| Arena header            | 16 bytes                                      |
+| Maximum decoded arena   | 65536 bytes                                   |
+| Maximum arena messages/parts | 256 / 4096                              |
+| Suggested default level | 1                                             |
 | Min compress, no dict   | 512 bytes                                     |
 | Min compress, with dict | 64 bytes                                      |
 | Max dictionary size     | 8 KiB                                         |
@@ -480,4 +551,4 @@ unbounded.
 - [Zstandard dictionary builder](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.h)
 - [CRIME attack](https://en.wikipedia.org/wiki/CRIME)
 - [BREACH attack](https://en.wikipedia.org/wiki/BREACH)
-- [`lz4+tcp://` RFC](../omq-lz4/RFC.md)
+- [`lz4+tcp://` RFC](lz4-rfc.md)

data/lib/omq/transport/zstd_tcp/codec.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "omq/transport/compression_arena"
+
 module OMQ
   module Transport
     module ZstdTcp
@@ -11,10 +13,12 @@ module OMQ
         TRAIN_MAX_SAMPLE_LEN   = 2048
         MIN_COMPRESS_NO_DICT   = 512
         MIN_COMPRESS_WITH_DICT = 64
+        ARENA_TARGET_BYTES    = 32 * 1024
 
-        NUL_PREAMBLE = ("\x00" * 4).b.freeze
+        NUL_PREAMBLE  = ("\x00" * 4).b.freeze
         ZSTD_MAGIC   = "\x28\xB5\x2F\xFD".b.freeze
         ZDICT_MAGIC  = "\x37\xA4\x30\xEC".b.freeze
+        ARENA_MAGIC  = "ZSTA".b.freeze
 
         USER_DICT_ID_RANGE = (32_768..(2**31 - 1)).freeze
 
@@ -62,6 +66,21 @@ module OMQ
         end
 
 
+        def compress_messages(messages)
+          CompressionArena.each_batch(messages, target: ARENA_TARGET_BYTES).map do |batch|
+            if batch.size == 1
+              compress_parts(batch.first)
+            else
+              batch.each { |parts| parts.each { |part| maybe_train!(part) } }
+              threshold = @send_dict_bytes ? MIN_COMPRESS_WITH_DICT : MIN_COMPRESS_NO_DICT
+              [CompressionArena.encode(batch, sentinel: ARENA_MAGIC, threshold: threshold) do |data|
+                @send_codec.compress(data)
+              end]
+            end
+          end
+        end
+
+
         def parse_frame_content_size(wire)
           return nil if wire.bytesize < 5
 
@@ -92,6 +111,33 @@ module OMQ
         end
 
 
+        # Zrip accepts unused trailing bytes. Walk the frame's block headers
+        # so each transport part contains exactly one complete Zstd frame.
+        def validate_frame!(wire)
+          raise ProtocolError, "truncated Zstd frame" if wire.bytesize < 5
+
+          descriptor = wire.getbyte(4)
+          single     = (descriptor & 0x20) != 0
+          fcs_bytes  = [single ? 1 : 0, 2, 4, 8][descriptor >> 6]
+          offset     = 5 + (single ? 0 : 1) + [0, 1, 2, 4][descriptor & 3] + fcs_bytes
+
+          loop do
+            raise ProtocolError, "truncated Zstd block header" if offset + 3 > wire.bytesize
+
+            header = wire.getbyte(offset) | (wire.getbyte(offset + 1) << 8) | (wire.getbyte(offset + 2) << 16)
+            type   = (header >> 1) & 3
+            raise ProtocolError, "reserved Zstd block type" if type == 3
+
+            offset += 3 + (type == 1 ? 1 : header >> 3)
+            raise ProtocolError, "truncated Zstd block" if offset > wire.bytesize
+            break unless (header & 1).zero?
+          end
+
+          offset += 4 unless (descriptor & 4).zero?
+          raise ProtocolError, "Zstd frame has trailing or missing bytes" unless offset == wire.bytesize
+        end
+
+
         private
 
 

data/lib/omq/transport/zstd_tcp/connection.rb

@@ -1,18 +1,19 @@
 # frozen_string_literal: true
 
-require "delegate"
+require "omq/transport/compressed_connection"
 
 module OMQ
   module Transport
     module ZstdTcp
-      class ZstdConnection < SimpleDelegator
+      class ZstdConnection < CompressedConnection
         # @return [Integer, nil] wire bytesize of the last received message
         #
         attr_reader :last_wire_size_in
 
 
         def initialize(conn, codec)
-          super(conn)
+          super(conn, max_message_size: codec.max_message_size,
+                arena_sentinel: Codec::ARENA_MAGIC, protocol_error: ProtocolError)
           @codec              = codec
           @dict_shipped       = false
           # FrameCodec is the decoder. Starts no-dict; when a dict
@@ -40,31 +41,22 @@ module OMQ
 
 
         def write_messages(messages)
-          compressed = messages.map { |parts| @codec.compress_parts(parts) }
+          compressed = @codec.compress_messages(messages)
           ship_dict!
           __getobj__.write_messages(compressed)
         end
 
 
-        def receive_message
-          loop do
-            parts   = __getobj__.receive_message
-            decoded = decode_parts(parts)
-            if decoded
-              @last_wire_size_in = parts.sum { |p| p.bytesize }
-              return decoded
-            end
-          end
-        end
-
+        private
 
-        def respond_to?(name, include_private = false)
-          return false if name == :write_wire
-          super
-        end
 
+        def decompress_arena(data, size)
+          unless data.start_with?(Codec::ZSTD_MAGIC) && @codec.parse_frame_content_size(data) == size
+            raise ProtocolError, "arena Frame_Content_Size mismatch"
+          end
 
-        private
+          decode_zstd_frame(data, size)
+        end
 
 
         def ship_dict!
@@ -84,6 +76,10 @@ module OMQ
           all_dicts = true
 
           parts.each do |wire|
+            if wire.start_with?(Codec::ZDICT_MAGIC) && parts.size != 1
+              raise ProtocolError, "dictionary must be a single-part message"
+            end
+
             plaintext = decode_part(wire, budget)
             if plaintext
               all_dicts = false
@@ -125,10 +121,12 @@ module OMQ
             raise ProtocolError, "declared FCS #{fcs} exceeds limit #{budget}"
           end
 
-          decompress_opts = budget ? { max_output_size: budget } : {}
-
+          @codec.validate_frame!(wire)
           codec = frame_has_dict_id?(wire) ? @recv_codec : @recv_no_dict_codec
-          codec.decompress(wire, **decompress_opts)
+          plaintext = codec.decompress(wire, max_output_size: fcs)
+          raise ProtocolError, "Zstd decoded size mismatch" unless plaintext.bytesize == fcs
+
+          plaintext
         rescue Zrip::DecompressError => e
           raise ProtocolError, "decompression failed: #{e.message}"
         rescue Zrip::MissingContentSizeError => e
@@ -146,6 +144,8 @@ module OMQ
 
 
         def install_recv_dict(wire)
+          raise ProtocolError, "second dictionary shipment on the same direction" if @recv_dict_bytes
+
           if wire.bytesize < 8
             raise ProtocolError, "dict frame too short"
           end
@@ -154,7 +154,11 @@ module OMQ
             raise ProtocolError, "dict exceeds #{Codec::MAX_DICT_SIZE} bytes"
           end
 
-          @recv_codec      = Zrip::FrameCodec.new(dict: wire.b)
+          @recv_codec = begin
+            Zrip::FrameCodec.new(dict: wire.b)
+          rescue RuntimeError => error
+            raise ProtocolError, "invalid Zstd dictionary: #{error.message}"
+          end
           @recv_dict_bytes = wire.b
         end
 

data/lib/omq/transport/zstd_tcp/transport.rb

@@ -29,6 +29,7 @@ module OMQ
         # @return [Listener]
         #
         def listener(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_security!(engine)
           validate_auto_dict!(auto_dict, dict)
           codec = codec_for(
             engine,
@@ -65,6 +66,7 @@ module OMQ
         # @return [Dialer]
         #
         def dialer(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_security!(engine)
           validate_auto_dict!(auto_dict, dict)
           codec = codec_for(
             engine,
@@ -109,6 +111,13 @@ module OMQ
         private
 
 
+        def validate_security!(engine)
+          if engine.options.mechanism&.encrypted?
+            raise ArgumentError, "compressed transports do not support encrypted mechanisms"
+          end
+        end
+
+
         # Returns the shared Codec for this engine (socket), creating one
         # on first call.  Stored in a WeakKeyMap keyed by engine — the
         # entry is automatically removed when the engine is GC'd.

data/lib/omq/transport/zstd_tcp.rb

@@ -18,7 +18,7 @@ require_relative "zstd_tcp/transport"
 module OMQ
   module Transport
     module ZstdTcp
-      class ProtocolError < StandardError; end
+      class ProtocolError < Protocol::ZMTP::Error; end
     end
   end
 end

data/lib/omq/zstd/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module Zstd
-    VERSION = "0.4.4"
+    VERSION = "0.5.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-zstd
 version: !ruby/object:Gem::Version
-  version: 0.4.4
+  version: 0.5.0
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,28 +15,28 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.28'
+        version: '0.29'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.28'
+        version: '0.29'
 - !ruby/object:Gem::Dependency
   name: zrip
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.1.1
+        version: 0.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.1.1
+        version: 0.2.0
 description: Experimental zstd+tcp:// endpoint support for OMQ with per-frame Zstd
   compression, bounded decompression, in-band dictionary shipping, and sender-side
   dictionary training.
@@ -74,7 +74,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.16
+rubygems_version: 4.0.20
 specification_version: 4
 summary: Experimental Zstd+TCP transport for OMQ
 test_files: []