omq-zstd 0.4.1 → 0.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0460be7b61085b54ba2e2de90033ed296524f08bfd56db2385ba3c69284713f1
-  data.tar.gz: 0061e9188f071929d68a19415fcd3207fae5f929cdddf80bd022e127007302e3
+  metadata.gz: 1fcea64306ec7603681d7b3350e7cdf2d22d43487be062e79cc91f51f64aacfb
+  data.tar.gz: ac5d8d88deb3f90ccf97be757d384612274f4d742b042b79bc949a47ea7d0d3c
 SHA512:
-  metadata.gz: 6730b00146a3ce45053466f14496b792b6cf619557555dafaed72debcd667c3dedf6a06200036114afb7adda81183c276d6212f42652a03b40ac62f13d2fa947
-  data.tar.gz: 46519bf5bf3e3f3f944404aa174591a49faa240946acce52f743cf0fcb5263a7a3a609d26667eb6bd2f1f38f406be22779c26ffad81e5f9c6f122f3379474536
+  metadata.gz: 8e5a04f9ffdb46d130af145fa7025c544cc1709e7711cd92180b0408923667e61d70098efc4679c7b0d2617a3acde40b91c9ecaa1b5abb9e56cadc4a34cd8294
+  data.tar.gz: 25ab87f733a74257a9b2d772d96a76be134b60804ce2796fa0a95710334eb0552f973fab4a7b04442f6d9cce8091da008c5ed17e6868e925b942fdc4dfefd0f6

data/README.md

@@ -4,13 +4,11 @@
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
 [![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
 
-> **Status:** Draft. Wire format may change before the first tagged release.
-
 Zstandard-compressed TCP transport for [OMQ](https://github.com/paddor/omq).
 Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
 compressed per-part with [Zstandard](https://github.com/facebook/zstd).
-Compression is intrinsic to the transport — no negotiation, no socket option,
-no payload changes. The ZMTP handshake itself runs over plain TCP; only
+Compression is intrinsic to the transport: no negotiation, no socket option,
+no payload changes. The ZMTP handshake runs over plain TCP; only
 post-handshake message parts are compressed.
 
 See [RFC.md](RFC.md) for the wire-format specification and
@@ -44,7 +42,7 @@ pull.receive  # => ["hello, compressed world"]
 ```
 
 Both peers must use the `zstd+tcp://` scheme. A `tcp://` peer cannot talk to
-a `zstd+tcp://` peer — they speak different transports.
+a `zstd+tcp://` peer. They speak different transports.
 
 ### Compression level
 
@@ -61,7 +59,7 @@ at any level the peer chose.
 ### Dictionaries
 
 Small messages don't compress well on their own. A shared Zstd dictionary
-trained on representative payloads gives 2–10× ratios on payloads in the
+trained on representative payloads gives 2-10x ratios on payloads in the
 dozens-to-hundreds-of-bytes range.
 
 **User-supplied dictionary** (out-of-band agreement):
@@ -75,11 +73,11 @@ The sender ships the dictionary to the receiver in-band as a one-shot
 single-part message prefixed with the dictionary sentinel
 (`37 A4 30 EC`), so the receiver does not need a copy on disk.
 
-**Auto-trained dictionary** (zero config — the default when no `dict:` is
+**Auto-trained dictionary** (zero config, the default when no `dict:` is
 passed): the sender collects up to 1000 samples or 100 KiB (whichever hits
-first), trains a dictionary, ships it inline, and switches to dictionary
-mode. Until then, payloads are compressed without a dictionary or sent
-plaintext when below the threshold.
+first), skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary,
+ships it inline, and switches to dictionary mode. Until then, payloads are
+compressed without a dictionary or sent plaintext when below the threshold.
 
 ### Compression thresholds
 
@@ -95,8 +93,8 @@ plaintext bytes).
 
 ### Security limits
 
-The receiver bounds decompression by the socket's own `max_message_size`
-— the same knob you'd use on a plain `tcp://` socket. It caps the
+The receiver bounds decompression by the socket's own `max_message_size`,
+the same knob you'd use on a plain `tcp://` socket. It caps the
 **total decompressed size of all parts in a single message**, not each
 part individually: the budget starts at `max_message_size` and shrinks
 as each part is decoded, so a message whose parts sum to more than the
@@ -111,10 +109,42 @@ ceiling on decompressed message size. Set a value that matches what
 your application would tolerate over plain `tcp://`.
 
 Independent of the message-size knob, the dictionary itself is capped at
-64 KiB (Zstd's recommended dictionary size range). A peer attempting to
-ship a larger dictionary, or send a message whose decompressed parts
-exceed `max_message_size`, drops the connection — `OMQ::SocketDeadError`
-surfaces on the next `receive`.
+8 KiB. A peer attempting to ship a larger dictionary, or send a message
+whose decompressed parts exceed `max_message_size`, drops the connection.
+`OMQ::SocketDeadError` surfaces on the next `receive`.
+
+## Wire format
+
+Every post-handshake ZMTP message part starts with a 4-byte sentinel:
+
+| Sentinel (hex) | Meaning |
+|---|---|
+| `00 00 00 00` | Uncompressed plaintext |
+| `28 B5 2F FD` | Zstandard-compressed frame |
+| `37 A4 30 EC` | Dictionary shipment |
+
+Compressed parts are standard Zstandard frames with `Frame_Content_Size`
+set in the header. The receiver uses FCS for budget enforcement before
+invoking the decoder. Any other leading 4 bytes close the connection.
+
+Dictionary shipments are single-part ZMTP messages consumed by the
+transport layer. They are not delivered to the application.
+
+## Constants
+
+| Constant | Value |
+|---|---|
+| Uncompressed sentinel | `00 00 00 00` |
+| Zstd frame sentinel | `28 B5 2F FD` (Zstandard frame magic) |
+| Dictionary sentinel | `37 A4 30 EC` |
+| Default level | -3 |
+| Min compress, no dict | 512 B |
+| Min compress, with dict | 64 B |
+| Max dictionary size | 8 KiB |
+| Train max samples | 1000 |
+| Train max bytes | 100 KiB |
+| Train max sample length | 2048 B |
+| Dictionary capacity | 2 KiB |
 
 ## When to use it
 
@@ -127,12 +157,12 @@ surfaces on the next `receive`.
 
 It is **not** worth it for:
 
-- `inproc://` or `ipc://` — irrelevant; there is no wire to shrink. Use
-  `zstd+tcp://` only on the connections that actually need it. Other
-  transports on the same socket are unaffected.
-- Already-compressed payloads (gzip, video, encrypted blobs) — the Zstd
+- `inproc://` or `ipc://`. No wire to shrink. Use `zstd+tcp://` only on
+  the connections that actually need it. Other transports on the same
+  socket are unaffected.
+- Already-compressed payloads (gzip, video, encrypted blobs). The Zstd
   pass adds CPU for no gain.
-- Latency-critical sub-microsecond paths — compression adds single-digit
+- Latency-critical sub-microsecond paths. Compression adds single-digit
   microseconds per kilobyte at low levels, but it is not free.
 
 ## How it works (in one paragraph)
@@ -140,7 +170,7 @@ It is **not** worth it for:
 `require "omq/zstd"` registers the `zstd+tcp` scheme on
 `OMQ::Engine.transports`. A `zstd+tcp` socket builds a per-engine
 `Codec` (one Zstd dictionary instance shared across all the socket's
-connections — fan-out compresses each part exactly once). Each accepted
+connections; fan-out compresses each part exactly once). Each accepted
 or dialed TCP connection is wrapped in `ZstdConnection`, a
 `SimpleDelegator` over the underlying ZMTP connection that intercepts
 `#send_message` / `#write_message` / `#receive_message`. Message parts

data/RFC.md

@@ -1,11 +1,13 @@
-# ZMTP over Zstd+TCP: Zstandard-Compressed TCP Transport for ZMTP
+# ZMTP-Zstd: Zstandard-Compressed TCP Transport for ZMTP
 
-| Field | Value |
+| Field    | Value                                              |
 |----------|----------------------------------------------------|
-| Status | Draft |
-| Editor | Patrik Wenger |
+| Status   | Draft                                              |
+| Editor   | Patrik Wenger                                      |
+| Scheme   | `zstd+tcp://`                                      |
 | Requires | [RFC 37/ZMTP 3.1](https://rfc.zeromq.org/spec/37/) |
 
+
 ## 1. Abstract
 
 This specification defines `zstd+tcp://`, a TCP transport for ZMTP 3.1
@@ -16,10 +18,19 @@ greeting and handshake proceed over raw TCP exactly as they would over
 is individually encoded with a 4-byte sentinel dispatch that
 distinguishes uncompressed plaintext, Zstandard-compressed frames, and
 dictionary shipments. No ZMTP properties, command frames, or
-negotiation are involved — compression is an intrinsic property of the
+negotiation are involved. Compression is an intrinsic property of the
 transport, like encryption is an intrinsic property of TLS.
 
-## 2. Motivation
+
+## 2. Language
+
+The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
+"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
+document are to be interpreted as described in
+[RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
+
+
+## 3. Motivation
 
 Zstandard at low compression levels encodes in single-digit microseconds
 per kilobyte, decompresses faster still, and on dictionary-trained
@@ -39,14 +50,14 @@ the transport. `zstd+tcp://` replaces it with a transport-level
 mechanism that any ZMTP application benefits from without changes to the
 payload.
 
-### 2.1 Why a transport scheme
+### 3.1 Why a transport scheme
 
 Compression could live at three layers. Each has a fatal flaw except the
 transport layer.
 
 **Socket-level wrapper** (too high). A wrapper above routing knows
 nothing about transports. It compresses local connections (pure
-overhead) and cannot act on new connections naturally — dictionary
+overhead) and cannot act on new connections naturally. Dictionary
 shipping requires per-connection state, but a wrapper only sees messages
 after routing has dispatched them. Reconnect handling requires hooking
 into connection lifecycle events that are awkward from outside.
@@ -61,13 +72,13 @@ connections.
 explicit in the endpoint URI. Only TCP connections get compressed. Local
 transports are unaffected even on the same socket. Dictionary lifetime
 matches connection lifetime naturally (new connection = new wrapper =
-re-ship dictionary). No negotiation is needed — both peers use
+re-ship dictionary). No negotiation is needed; both peers use
 `zstd+tcp://`. The codec is socket-wide (shared across connections), so
 fan-out patterns compress once and reuse the result.
 
-### 2.2 Why not negotiate
+### 3.2 Why not negotiate
 
-ZMTP 3.1 already supports unknown READY properties — an unaware peer
+ZMTP 3.1 already supports unknown READY properties. An unaware peer
 silently ignores them. A negotiation-based design could fall back to
 plaintext when the peer does not understand compression. But this
 introduces complexity (profile matching, asymmetric per-direction state,
@@ -76,7 +87,7 @@ deployment decision, not a runtime discovery. Both peers are configured
 to use `zstd+tcp://` or they are not. The transport scheme approach
 eliminates the entire negotiation surface and its edge cases.
 
-### 2.3 Why Zstandard
+### 3.3 Why Zstandard
 
 Zstandard at low levels matches LZ4 on encode latency, beats it on
 decompression speed and ratio at every realistic ZMQ payload size, and
@@ -85,9 +96,10 @@ particularly important for fan-out patterns (PUB/SUB, RADIO/DISH): the
 publisher pays one compress, every subscriber pays decompress, so
 per-subscriber CPU dominates the total budget.
 
-## 3. Goals and Non-goals
 
-### 3.1 Goals
+## 4. Goals and Non-goals
+
+### 4.1 Goals
 
 - Transparent to application code: send/receive operations see plaintext.
 - Per-part sender decision: opt out for short or incompressible parts.
@@ -98,50 +110,52 @@ per-subscriber CPU dominates the total budget.
 - No ZMTP-level negotiation, no new READY properties, no new command
   frames.
 
-### 3.2 Non-goals
+### 4.2 Non-goals
 
 - New ZMTP mechanism, new socket type, new greeting, new frame flag bit.
 - Compression of the ZMTP greeting or command frames (READY, SUBSCRIBE,
   PING, PONG, ...).
-- Application to non-TCP transports (`inproc://` is zero-copy —
+- Application to non-TCP transports (`inproc://` is zero-copy;
   compression is pure overhead; `ipc://` rarely benefits).
 - Replacing or weakening CurveZMQ or any other security mechanism.
-  See Sec. 8.
+  See Sec. 9.
 - Streaming / context-takeover compression. Each part is decodable in
   isolation with no dependency on a previous part's LZ77 history.
 
-## 4. Terminology
 
-| Term | Meaning |
-|---------------------|---------------------------------------------------------------------------|
-| Part | One ZMTP message frame body. A multipart message has multiple parts.      |
-| Sentinel            | The first 4 bytes of a post-handshake part on the wire (Sec. 5.1).       |
-| Uncompressed part   | A wire part whose sentinel is `00 00 00 00`.                              |
-| Compressed part     | A wire part whose first 4 bytes are the Zstandard magic `28 B5 2F FD`.   |
-| Dictionary part     | A wire part whose first 4 bytes are `37 A4 30 EC` (Sec. 6).              |
-| Dictionary message  | A single-part ZMTP message consisting of exactly one dictionary part.     |
+## 5. Terminology
+
+| Term                | Meaning                                                              |
+|---------------------|----------------------------------------------------------------------|
+| Part                | One ZMTP message frame body. A multipart message has multiple parts. |
+| Sentinel            | The first 4 bytes of a post-handshake part on the wire (Sec. 6.1).   |
+| Uncompressed part   | A wire part whose sentinel is `00 00 00 00`.                         |
+| Compressed part     | A wire part whose first 4 bytes are the Zstandard magic `28 B5 2F FD`. |
+| Dictionary part     | A wire part whose first 4 bytes are `37 A4 30 EC` (Sec. 7).         |
+| Dictionary message  | A single-part ZMTP message consisting of exactly one dictionary part. |
+
 
-## 5. Part Encoding
+## 6. Part Encoding
 
 After the ZMTP handshake completes, every message part on the wire is
 individually encoded. The ZMTP MORE flag is carried on the wire frame
-header as normal. Multipart messages are encoded part by part — each
+header as normal. Multipart messages are encoded part by part; each
 part is independent.
 
-### 5.1 Sentinel dispatch
+### 6.1 Sentinel dispatch
 
 The first 4 bytes of each wire part determine how it is decoded.
 
-| Sentinel (hex)   | Meaning                                                         |
-|------------------|-----------------------------------------------------------------|
-| `00 00 00 00`    | Uncompressed plaintext (Sec. 5.3)                               |
-| `28 B5 2F FD`    | Zstandard compressed frame (Sec. 5.4)                           |
-| `37 A4 30 EC`    | Dictionary shipment (Sec. 6)                                    |
+| Sentinel (hex)   | Meaning                              |
+|------------------|--------------------------------------|
+| `00 00 00 00`    | Uncompressed plaintext (Sec. 6.3)    |
+| `28 B5 2F FD`    | Zstandard compressed frame (Sec. 6.4)|
+| `37 A4 30 EC`    | Dictionary shipment (Sec. 7)         |
 
 All other 4-byte values are reserved. A receiver that encounters an
-unknown sentinel MUST drop the connection with an error.
+unknown sentinel MUST close the connection.
 
-### 5.2 Compression level
+### 6.2 Compression level
 
 The default compression level is **-3** (Zstandard fast strategy). At
 this level the encoder cost is in the low single-digit microseconds per
@@ -149,11 +163,11 @@ kilobyte, and the achieved ratio is within a few percent of level 3 once
 a dictionary is in play.
 
 The compression level is a sender choice and is not communicated on the
-wire — the receiver decodes any valid Zstandard frame regardless of the
+wire. The receiver decodes any valid Zstandard frame regardless of the
 level used to encode it. Implementations SHOULD expose the level as a
 configurable parameter.
 
-### 5.3 Uncompressed sentinel `00 00 00 00`
+### 6.3 Uncompressed sentinel `00 00 00 00`
 
 ```
 +------------------+-------------------+
@@ -169,7 +183,7 @@ without an extra flag bit in the ZMTP frame header.
 Four zero bytes cannot collide with a valid Zstandard frame magic or the
 dictionary sentinel, so no ambiguity arises.
 
-### 5.4 Compressed Zstandard frame
+### 6.4 Compressed Zstandard frame
 
 ```
 +------------------+
@@ -178,15 +192,15 @@ dictionary sentinel, so no ambiguity arises.
 +------------------+
 ```
 
-The wire part IS the Zstandard frame — its first 4 bytes are the
+The wire part IS the Zstandard frame. Its first 4 bytes are the
 standard Zstandard frame magic `28 B5 2F FD`. No additional framing is
 added.
 
 The sender MUST configure the encoder to write the `Frame_Content_Size`
-field in the Zstandard frame header (RFC 8878 §3.1.1.1.2). This field
-is required for the receiver's budget enforcement (Sec. 5.6).
+field in the Zstandard frame header (RFC 8878 Sec. 3.1.1.1.2). This
+field is required for the receiver's budget enforcement (Sec. 6.6).
 
-### 5.5 Sender rules
+### 6.5 Sender rules
 
 For each outgoing message part, the sender proceeds as follows:
 
@@ -203,7 +217,7 @@ For each outgoing message part, the sender proceeds as follows:
 
 3. Otherwise, run the Zstandard encoder. The encoder MUST write the
    `Frame_Content_Size` field. If the compressed output's size is
-   ≥ `plaintext_size - 4` (net saving ≤ 0 after accounting for the
+   >= `plaintext_size - 4` (net saving <= 0 after accounting for the
    4-byte sentinel of the uncompressed alternative), prepend
    `00 00 00 00` and emit the plaintext instead. Otherwise emit the
    Zstandard frame as-is.
@@ -213,41 +227,42 @@ For each outgoing message part, the sender proceeds as follows:
    MUST still prepend `00 00 00 00` to avoid sentinel ambiguity.
    Step 2 and step 3's fallback path already guarantee this.
 
-### 5.6 Receiver rules
+### 6.6 Receiver rules
 
 For each incoming wire part, the receiver proceeds as follows:
 
 1. Read the first 4 bytes as the sentinel. If the part is shorter than
-   4 bytes, drop the connection with an error.
+   4 bytes, close the connection.
 
 2. Sentinel `00 00 00 00`: the remaining `N - 4` bytes are plaintext.
    Return them.
 
 3. Sentinel `28 B5 2F FD`: the entire wire part is a Zstandard frame.
    - Read the `Frame_Content_Size` field from the Zstandard header. If
-     the field is absent, drop the connection with an error.
+     the field is absent, close the connection.
    - If the connection enforces a maximum message size, add this part's
      declared content size to the running decompressed total for the
      current multipart message (parts chained by the ZMTP MORE flag).
-     If the running total would exceed the maximum, drop the connection
-     with an error without invoking the decoder.
+     If the running total would exceed the maximum, close the connection
+     without invoking the decoder.
    - Invoke the decoder in a bounded mode that aborts if it would write
      more bytes than `Frame_Content_Size` declared. On such an abort,
-     drop the connection with an error.
+     close the connection.
    - Return the decompressed plaintext.
 
-4. Sentinel `37 A4 30 EC`: dictionary shipment. See Sec. 6.
+4. Sentinel `37 A4 30 EC`: dictionary shipment. See Sec. 7.
 
-5. Any other sentinel: drop the connection with an error.
+5. Any other sentinel: close the connection.
 
 The maximum message size always refers to the **decompressed** plaintext
 summed across all parts of a multipart message. A multipart message
 whose total wire length is small but whose total decompressed size
 exceeds the limit MUST be rejected before decoder invocation.
 
-## 6. Dictionary Shipment
 
-### 6.1 Dictionary message format
+## 7. Dictionary Shipment
+
+### 7.1 Dictionary message format
 
 A dictionary is shipped as a **single-part ZMTP message** (no MORE flag)
 whose body begins with the dictionary sentinel:
@@ -266,40 +281,40 @@ with the Zstandard frame magic and the uncompressed sentinel.
 The remaining `D` bytes are the raw dictionary as it should be passed
 to the Zstandard decoder's dictionary-load operation.
 
-### 6.2 Constraints
+### 7.2 Constraints
 
 - A dictionary message MUST be a single-part ZMTP message (MORE flag
   not set on the frame header). A dictionary sentinel in a multipart
   message's non-final or non-only part is a protocol error.
 
-- A dictionary message MUST NOT exceed **64 KiB** total (sentinel +
+- A dictionary message MUST NOT exceed **8 KiB** total (sentinel +
   dictionary bytes). A receiver that receives a dictionary message
-  larger than 64 KiB MUST drop the connection with an error.
+  larger than 8 KiB MUST close the connection.
 
 - A sender MUST send at most **one** dictionary message per direction
   per connection. A receiver that receives a second dictionary message
-  on the same connection MUST drop the connection with an error.
+  on the same connection MUST close the connection.
 
 - A dictionary message MUST be sent BEFORE any compressed part that
   references the dictionary. In practice this means the sender ships
   the dictionary before (or immediately after training triggers during)
   the first compressed write that would benefit from it.
 
-### 6.3 Receiver handling
+### 7.3 Receiver handling
 
 When the receiver encounters a dictionary part:
 
-1. Validate the constraints in Sec. 6.2.
+1. Validate the constraints in Sec. 7.2.
 2. Strip the 4-byte sentinel.
 3. Install the remaining bytes as the decompression dictionary for this
    connection.
-4. Discard the message — it is not delivered to the application.
+4. Discard the message. It is not delivered to the application.
 
 If all parts of a ZMTP message are dictionary parts (which is always
 the case, since dictionary messages are single-part), the receiver
 loops to receive the next message.
 
-### 6.4 Dictionary scope
+### 7.4 Dictionary scope
 
 The dictionary a sender ships applies to a single direction of a single
 connection. Each peer may independently ship its own dictionary for its
@@ -309,7 +324,7 @@ nothing (or uncompressed traffic) back.
 
 The sender's dictionary is typically socket-wide: trained once from
 early traffic across all connections and reused. But this is an
-implementation choice — the wire protocol carries no dictionary identity
+implementation choice. The wire protocol carries no dictionary identity
 or scope metadata.
 
 An implementation MAY pool training samples and share the resulting
@@ -319,21 +334,21 @@ multiple `zstd+tcp://` endpoints: samples from one endpoint accelerate
 training for all of them, and newly opened connections benefit from a
 dictionary trained by their predecessors. Connections that were
 configured with an explicit out-of-band dictionary MUST NOT participate
-in shared training — they use their own dictionary independently.
+in shared training; they use their own dictionary independently.
 
-### 6.5 Automatic dictionary training
+### 7.5 Automatic dictionary training
 
 A sender MAY train a dictionary automatically from early traffic:
 
 1. Buffer plaintext samples from the first messages. Samples larger
-   than **1024 bytes** SHOULD be skipped — dictionaries primarily
+   than **2048 bytes** SHOULD be skipped; dictionaries primarily
    benefit small frames.
 2. When the buffer reaches **1000 samples** OR **100 KiB** of
    plaintext (whichever comes first), train a Zstandard dictionary from
    the buffered samples and discard the buffer.
 3. The recommended dictionary capacity (training target size) is
-   **8 KiB**.
-4. Ship the trained dictionary via a dictionary message (Sec. 6.1) on
+   **2 KiB**.
+4. Ship the trained dictionary via a dictionary message (Sec. 7.1) on
    every connection, before any compressed part that uses it.
 5. Switch to dictionary-bound compression for all subsequent parts.
 
@@ -341,16 +356,17 @@ If training fails (the sample set was too small or too uniform), the
 sender MUST stay in no-dictionary mode for the rest of the socket's
 lifetime. It MUST NOT retry training.
 
-### 6.6 Dictionary ID
+### 7.6 Dictionary ID
 
 Auto-trained dictionaries SHOULD be patched with a random dictionary ID
 in the Zstandard user range (32768 to 2^31 - 1) to avoid collisions
 with Zstandard's built-in dictionary IDs. Out-of-band dictionaries
 retain whatever dictionary ID they were created with.
 
-## 7. ZMTP Interaction
 
-### 7.1 Greeting and handshake
+## 8. ZMTP Interaction
+
+### 8.1 Greeting and handshake
 
 The ZMTP greeting and security mechanism handshake proceed over raw TCP
 exactly as specified by RFC 37. `zstd+tcp://` does not modify the
@@ -358,28 +374,29 @@ greeting, mechanism, READY properties, or any command frames. The
 compression layer activates only after the handshake is complete and the
 connection is ready for message traffic.
 
-### 7.2 Command frames
+### 8.2 Command frames
 
 ZMTP command frames (READY, SUBSCRIBE, CANCEL, JOIN, LEAVE, PING,
 PONG) are never compressed. They are sent and received as standard ZMTP
 command frames. Only message frames (the COMMAND bit not set in the
 frame header) are subject to sentinel-dispatched encoding.
 
-### 7.3 Socket type compatibility
+### 8.3 Socket type compatibility
 
 `zstd+tcp://` is compatible with all ZMTP socket types. The socket type
 negotiation in the READY handshake is unaffected.
 
-### 7.4 Peer requirement
+### 8.4 Peer requirement
 
 Both peers of a connection MUST use `zstd+tcp://`. There is no
 fallback to plaintext TCP and no negotiation. A `zstd+tcp://` peer
 connecting to a plain `tcp://` peer (or vice versa) will see garbled
 data or sentinel errors and the connection will fail.
 
-## 8. Security Considerations
 
-### 8.1 Compression combined with encryption (CRIME / BREACH)
+## 9. Security Considerations
+
+### 9.1 Compression combined with encryption (CRIME / BREACH)
 
 Combining length-revealing compression with a secure channel that
 carries attacker-influenced plaintext enables CRIME- and BREACH-style
@@ -392,30 +409,30 @@ encrypted tunnel when the plaintext contains attacker-controlled
 content. Deployments that accept this risk MUST do so with explicit
 opt-in.
 
-### 8.2 Length side-channel
+### 9.2 Length side-channel
 
 Compression makes the wire length of a part depend on its content. An
 on-path observer can learn something about the plaintext from the
 compressed length alone. Deployments that care about traffic analysis
 MUST NOT rely on `zstd+tcp://` to hide payload shape.
 
-### 8.3 Dictionary contents
+### 9.3 Dictionary contents
 
 When auto-training is enabled, the receiver loads dictionary bytes
 chosen by the peer. The Zstandard reference dictionary loader is
 hardened against malformed inputs, but implementations MUST enforce the
-64 KiB cap on dictionary messages (Sec. 6.2) and SHOULD NOT cache
+8 KiB cap on dictionary messages (Sec. 7.2) and SHOULD NOT cache
 received dictionaries across connections.
 
-### 8.4 Decompression bombs
+### 9.4 Decompression bombs
 
 A small compressed frame can decompress to many megabytes of plaintext.
-The receiver rules in Sec. 5.6 mitigate this:
+The receiver rules in Sec. 6.6 mitigate this:
 
 1. Every compressed part MUST carry `Frame_Content_Size`. The receiver
    checks the declared total against the maximum message size before
    invoking the decoder, so a bomb is rejected on its header alone.
-2. The decoder is invoked in bounded mode — it aborts if it would write
+2. The decoder is invoked in bounded mode. It aborts if it would write
    more bytes than declared. A peer that lies in the header cannot
    expand a part past its declared size.
 
@@ -423,30 +440,30 @@ Implementations SHOULD set a conservative maximum message size on
 `zstd+tcp://` connections even if they would otherwise leave it
 unbounded.
 
-## 9. Constants
 
-```
-SENTINEL_UNCOMPRESSED   = 00 00 00 00   (4 bytes)
-SENTINEL_ZSTD_FRAME     = 28 B5 2F FD   (4 bytes, Zstandard frame magic)
-SENTINEL_ZSTD_DICT      = 37 A4 30 EC   (4 bytes)
+## 10. Constants
 
-DEFAULT_LEVEL           = -3
+| Constant                | Value                                         |
+|-------------------------|-----------------------------------------------|
+| Uncompressed sentinel   | `00 00 00 00`                                 |
+| Zstd frame sentinel     | `28 B5 2F FD` (Zstandard frame magic)         |
+| Dictionary sentinel     | `37 A4 30 EC`                                 |
+| Default level           | -3                                            |
+| Min compress, no dict   | 512 bytes                                     |
+| Min compress, with dict | 64 bytes                                      |
+| Max dictionary size     | 8 KiB                                         |
+| Train max samples       | 1000                                          |
+| Train max bytes         | 100 KiB                                       |
+| Train max sample length | 2048 bytes                                    |
+| Dictionary capacity     | 2 KiB                                         |
 
-MIN_COMPRESS_NO_DICT    = 512 bytes
-MIN_COMPRESS_WITH_DICT  = 64 bytes
-
-MAX_DICT_SIZE           = 64 KiB
-
-TRAIN_MAX_SAMPLES       = 1000
-TRAIN_MAX_BYTES         = 100 KiB
-TRAIN_MAX_SAMPLE_LEN    = 1024 bytes
-DICT_CAPACITY           = 8 KiB
-```
 
-## 10. References
+## 11. References
 
-- [RFC 37 / ZMTP 3.1](https://rfc.zeromq.org/spec/37/) — underlying wire protocol
-- [RFC 8878 — Zstandard Compression Data Format](https://datatracker.ietf.org/doc/html/rfc8878)
+- [RFC 37/ZMTP 3.1](https://rfc.zeromq.org/spec/37/)
+- [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119)
+- [RFC 8878: Zstandard Compression Data Format](https://datatracker.ietf.org/doc/html/rfc8878)
 - [Zstandard dictionary builder](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.h)
-- [CRIME attack](https://en.wikipedia.org/wiki/CRIME) — compression side-channel on TLS
-- [BREACH attack](https://en.wikipedia.org/wiki/BREACH) — HTTP-layer variant
+- [CRIME attack](https://en.wikipedia.org/wiki/CRIME)
+- [BREACH attack](https://en.wikipedia.org/wiki/BREACH)
+- [`lz4+tcp://` RFC](../omq-lz4/RFC.md)

data/lib/omq/transport/zstd_tcp/codec.rb

@@ -4,11 +4,11 @@ module OMQ
   module Transport
     module ZstdTcp
       class Codec
-        MAX_DICT_SIZE          = 64 * 1024
-        DICT_CAPACITY          = 8 * 1024
+        MAX_DICT_SIZE          = 8 * 1024
+        DICT_CAPACITY          = 2 * 1024
         TRAIN_MAX_SAMPLES      = 1000
         TRAIN_MAX_BYTES        = 100 * 1024
-        TRAIN_MAX_SAMPLE_LEN   = 1024
+        TRAIN_MAX_SAMPLE_LEN   = 2048
         MIN_COMPRESS_NO_DICT   = 512
         MIN_COMPRESS_WITH_DICT = 64
 

data/lib/omq/zstd/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module Zstd
-    VERSION = "0.4.1"
+    VERSION = "0.4.2"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-zstd
 version: !ruby/object:Gem::Version
-  version: 0.4.1
+  version: 0.4.2
 platform: ruby
 authors:
 - Patrik Wenger
@@ -73,7 +73,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.6
+rubygems_version: 4.0.10
 specification_version: 4
 summary: Zstd+TCP transport for OMQ
 test_files: []