omq-zstd 0.4.1 → 0.4.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (9) hide show
  1. checksums.yaml +4 -4
  2. data/DESIGN.md +2 -2
  3. data/README.md +60 -23
  4. data/RFC.md +141 -115
  5. data/lib/omq/transport/zstd_tcp/codec.rb +14 -16
  6. data/lib/omq/transport/zstd_tcp/connection.rb +18 -14
  7. data/lib/omq/transport/zstd_tcp.rb +1 -1
  8. data/lib/omq/zstd/version.rb +1 -1
  9. metadata +13 -12
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 0460be7b61085b54ba2e2de90033ed296524f08bfd56db2385ba3c69284713f1
4
- data.tar.gz: 0061e9188f071929d68a19415fcd3207fae5f929cdddf80bd022e127007302e3
3
+ metadata.gz: 5a3ed977d7fa2b57910a209fd3db1ca683634a9ccc2d299cd3dd0f95c0030d93
4
+ data.tar.gz: 4d439082a1142c335538c3ea3698cc629e6dc9fa406335386171d6457ad2a9e6
5
5
  SHA512:
6
- metadata.gz: 6730b00146a3ce45053466f14496b792b6cf619557555dafaed72debcd667c3dedf6a06200036114afb7adda81183c276d6212f42652a03b40ac62f13d2fa947
7
- data.tar.gz: 46519bf5bf3e3f3f944404aa174591a49faa240946acce52f743cf0fcb5263a7a3a609d26667eb6bd2f1f38f406be22779c26ffad81e5f9c6f122f3379474536
6
+ metadata.gz: a68d93c3c116a9fe508844058515f74733a674c6a8a085149b8bff2a8cfb089793509a5d3aed54cc56c3730ecd8386e6eb12ac500cad734e58d8976b6cc674bd
7
+ data.tar.gz: 9eb6751a7682a0ebefd9680b9c48f8a1f6c31041e2f276e87064054d6d7094145a8e6a4776579ea33b9d4c10eacd6d4ce6817c6c4cc3e0d4ec1401ecd659a01c
data/DESIGN.md CHANGED
@@ -67,7 +67,7 @@ One Codec per Dialer or Listener. Shared across all connections of
67
67
  that endpoint. Owns:
68
68
 
69
69
  - **Compression**: `#compress_parts(parts)` with identity cache
70
- - **Training**: sample collection, `RZstd::Dictionary.train`, dict ID patching
70
+ - **Training**: sample collection, `Zrip::DictTrainer`, dict ID patching
71
71
  - **Send dict**: `#send_dict_bytes` — the trained or user-supplied dict bytes
72
72
 
73
73
  ### ZstdConnection (per-connection)
@@ -124,7 +124,7 @@ end
124
124
  - **Sample size cap**: frames > 1024 bytes are skipped (dictionaries primarily benefit small frames)
125
125
  - **Dict capacity**: 8 KiB (conservative; Zstd recommends ~100:1 sample-to-dict ratio)
126
126
  - **Dict ID patching**: auto-trained dicts get a random ID in the user range (32768..2^31-1) to avoid collisions with Zstd's built-in dict IDs
127
- - **Training failure**: if `RZstd::Dictionary.train` raises, training is disabled permanently for the socket. No retry.
127
+ - **Training failure**: if `Zrip::DictTrainer#train` raises, training is disabled permanently for the socket. No retry.
128
128
 
129
129
  ## Frame dispatch
130
130
 
data/README.md CHANGED
@@ -2,15 +2,20 @@
2
2
 
3
3
  [![Gem Version](https://img.shields.io/gem/v/omq-zstd?color=e9573f)](https://rubygems.org/gems/omq-zstd)
4
4
  [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
5
- [![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
5
+ [![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%204.0-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
6
6
 
7
- > **Status:** Draft. Wire format may change before the first tagged release.
7
+ Experimental Zstandard-compressed TCP transport for
8
+ [OMQ](https://github.com/zeromq/omq.rb).
9
+
10
+ Use [`omq-lz4`](../omq-lz4) for new compressed TCP work. OMQ.rs removed
11
+ `zstd+tcp://` to reduce transport complexity after lz4rip gained
12
+ dictionary training. This gem remains for research, comparison, and
13
+ cases where zstd ratio matters more than transport simplicity.
8
14
 
9
- Zstandard-compressed TCP transport for [OMQ](https://github.com/paddor/omq).
10
15
  Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
11
16
  compressed per-part with [Zstandard](https://github.com/facebook/zstd).
12
- Compression is intrinsic to the transport — no negotiation, no socket option,
13
- no payload changes. The ZMTP handshake itself runs over plain TCP; only
17
+ Compression is intrinsic to the transport: no negotiation, no socket option,
18
+ no payload changes. The ZMTP handshake runs over plain TCP. Only
14
19
  post-handshake message parts are compressed.
15
20
 
16
21
  See [RFC.md](RFC.md) for the wire-format specification and
@@ -44,7 +49,7 @@ pull.receive # => ["hello, compressed world"]
44
49
  ```
45
50
 
46
51
  Both peers must use the `zstd+tcp://` scheme. A `tcp://` peer cannot talk to
47
- a `zstd+tcp://` peer — they speak different transports.
52
+ a `zstd+tcp://` peer. They speak different transports.
48
53
 
49
54
  ### Compression level
50
55
 
@@ -61,7 +66,7 @@ at any level the peer chose.
61
66
  ### Dictionaries
62
67
 
63
68
  Small messages don't compress well on their own. A shared Zstd dictionary
64
- trained on representative payloads gives 2–10× ratios on payloads in the
69
+ trained on representative payloads gives 2-10x ratios on payloads in the
65
70
  dozens-to-hundreds-of-bytes range.
66
71
 
67
72
  **User-supplied dictionary** (out-of-band agreement):
@@ -75,11 +80,11 @@ The sender ships the dictionary to the receiver in-band as a one-shot
75
80
  single-part message prefixed with the dictionary sentinel
76
81
  (`37 A4 30 EC`), so the receiver does not need a copy on disk.
77
82
 
78
- **Auto-trained dictionary** (zero config — the default when no `dict:` is
83
+ **Auto-trained dictionary** (zero config, the default when no `dict:` is
79
84
  passed): the sender collects up to 1000 samples or 100 KiB (whichever hits
80
- first), trains a dictionary, ships it inline, and switches to dictionary
81
- mode. Until then, payloads are compressed without a dictionary or sent
82
- plaintext when below the threshold.
85
+ first), skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary,
86
+ ships it inline, and switches to dictionary mode. Until then, payloads are
87
+ compressed without a dictionary or sent plaintext when below the threshold.
83
88
 
84
89
  ### Compression thresholds
85
90
 
@@ -95,8 +100,8 @@ plaintext bytes).
95
100
 
96
101
  ### Security limits
97
102
 
98
- The receiver bounds decompression by the socket's own `max_message_size`
99
- — the same knob you'd use on a plain `tcp://` socket. It caps the
103
+ The receiver bounds decompression by the socket's own `max_message_size`,
104
+ the same knob you'd use on a plain `tcp://` socket. It caps the
100
105
  **total decompressed size of all parts in a single message**, not each
101
106
  part individually: the budget starts at `max_message_size` and shrinks
102
107
  as each part is decoded, so a message whose parts sum to more than the
@@ -111,10 +116,42 @@ ceiling on decompressed message size. Set a value that matches what
111
116
  your application would tolerate over plain `tcp://`.
112
117
 
113
118
  Independent of the message-size knob, the dictionary itself is capped at
114
- 64 KiB (Zstd's recommended dictionary size range). A peer attempting to
115
- ship a larger dictionary, or send a message whose decompressed parts
116
- exceed `max_message_size`, drops the connection — `OMQ::SocketDeadError`
117
- surfaces on the next `receive`.
119
+ 8 KiB. A peer attempting to ship a larger dictionary, or send a message
120
+ whose decompressed parts exceed `max_message_size`, drops the connection.
121
+ `OMQ::SocketDeadError` surfaces on the next `receive`.
122
+
123
+ ## Wire format
124
+
125
+ Every post-handshake ZMTP message part starts with a 4-byte sentinel:
126
+
127
+ | Sentinel (hex) | Meaning |
128
+ |---|---|
129
+ | `00 00 00 00` | Uncompressed plaintext |
130
+ | `28 B5 2F FD` | Zstandard-compressed frame |
131
+ | `37 A4 30 EC` | Dictionary shipment |
132
+
133
+ Compressed parts are standard Zstandard frames with `Frame_Content_Size`
134
+ set in the header. The receiver uses FCS for budget enforcement before
135
+ invoking the decoder. Any other leading 4 bytes close the connection.
136
+
137
+ Dictionary shipments are single-part ZMTP messages consumed by the
138
+ transport layer. They are not delivered to the application.
139
+
140
+ ## Constants
141
+
142
+ | Constant | Value |
143
+ |---|---|
144
+ | Uncompressed sentinel | `00 00 00 00` |
145
+ | Zstd frame sentinel | `28 B5 2F FD` (Zstandard frame magic) |
146
+ | Dictionary sentinel | `37 A4 30 EC` |
147
+ | Default level | -3 |
148
+ | Min compress, no dict | 512 B |
149
+ | Min compress, with dict | 64 B |
150
+ | Max dictionary size | 8 KiB |
151
+ | Train max samples | 1000 |
152
+ | Train max bytes | 100 KiB |
153
+ | Train max sample length | 2048 B |
154
+ | Dictionary capacity | 2 KiB |
118
155
 
119
156
  ## When to use it
120
157
 
@@ -127,12 +164,12 @@ surfaces on the next `receive`.
127
164
 
128
165
  It is **not** worth it for:
129
166
 
130
- - `inproc://` or `ipc://` — irrelevant; there is no wire to shrink. Use
131
- `zstd+tcp://` only on the connections that actually need it. Other
132
- transports on the same socket are unaffected.
133
- - Already-compressed payloads (gzip, video, encrypted blobs) — the Zstd
167
+ - `inproc://` or `ipc://`. No wire to shrink. Use `zstd+tcp://` only on
168
+ the connections that actually need it. Other transports on the same
169
+ socket are unaffected.
170
+ - Already-compressed payloads (gzip, video, encrypted blobs). The Zstd
134
171
  pass adds CPU for no gain.
135
- - Latency-critical sub-microsecond paths — compression adds single-digit
172
+ - Latency-critical sub-microsecond paths. Compression adds single-digit
136
173
  microseconds per kilobyte at low levels, but it is not free.
137
174
 
138
175
  ## How it works (in one paragraph)
@@ -140,7 +177,7 @@ It is **not** worth it for:
140
177
  `require "omq/zstd"` registers the `zstd+tcp` scheme on
141
178
  `OMQ::Engine.transports`. A `zstd+tcp` socket builds a per-engine
142
179
  `Codec` (one Zstd dictionary instance shared across all the socket's
143
- connections — fan-out compresses each part exactly once). Each accepted
180
+ connections. Fan-out compresses each part exactly once). Each accepted
144
181
  or dialed TCP connection is wrapped in `ZstdConnection`, a
145
182
  `SimpleDelegator` over the underlying ZMTP connection that intercepts
146
183
  `#send_message` / `#write_message` / `#receive_message`. Message parts
data/RFC.md CHANGED
@@ -1,11 +1,13 @@
1
- # ZMTP over Zstd+TCP: Zstandard-Compressed TCP Transport for ZMTP
1
+ # ZMTP-Zstd: Zstandard-Compressed TCP Transport for ZMTP
2
2
 
3
- | Field | Value |
3
+ | Field | Value |
4
4
  |----------|----------------------------------------------------|
5
- | Status | Draft |
6
- | Editor | Patrik Wenger |
5
+ | Status | Draft |
6
+ | Editor | Patrik Wenger |
7
+ | Scheme | `zstd+tcp://` |
7
8
  | Requires | [RFC 37/ZMTP 3.1](https://rfc.zeromq.org/spec/37/) |
8
9
 
10
+
9
11
  ## 1. Abstract
10
12
 
11
13
  This specification defines `zstd+tcp://`, a TCP transport for ZMTP 3.1
@@ -16,10 +18,19 @@ greeting and handshake proceed over raw TCP exactly as they would over
16
18
  is individually encoded with a 4-byte sentinel dispatch that
17
19
  distinguishes uncompressed plaintext, Zstandard-compressed frames, and
18
20
  dictionary shipments. No ZMTP properties, command frames, or
19
- negotiation are involved — compression is an intrinsic property of the
21
+ negotiation are involved. Compression is an intrinsic property of the
20
22
  transport, like encryption is an intrinsic property of TLS.
21
23
 
22
- ## 2. Motivation
24
+
25
+ ## 2. Language
26
+
27
+ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
28
+ "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
29
+ document are to be interpreted as described in
30
+ [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119).
31
+
32
+
33
+ ## 3. Motivation
23
34
 
24
35
  Zstandard at low compression levels encodes in single-digit microseconds
25
36
  per kilobyte, decompresses faster still, and on dictionary-trained
@@ -39,14 +50,14 @@ the transport. `zstd+tcp://` replaces it with a transport-level
39
50
  mechanism that any ZMTP application benefits from without changes to the
40
51
  payload.
41
52
 
42
- ### 2.1 Why a transport scheme
53
+ ### 3.1 Why a transport scheme
43
54
 
44
55
  Compression could live at three layers. Each has a fatal flaw except the
45
56
  transport layer.
46
57
 
47
58
  **Socket-level wrapper** (too high). A wrapper above routing knows
48
59
  nothing about transports. It compresses local connections (pure
49
- overhead) and cannot act on new connections naturally — dictionary
60
+ overhead) and cannot act on new connections naturally. Dictionary
50
61
  shipping requires per-connection state, but a wrapper only sees messages
51
62
  after routing has dispatched them. Reconnect handling requires hooking
52
63
  into connection lifecycle events that are awkward from outside.
@@ -61,13 +72,13 @@ connections.
61
72
  explicit in the endpoint URI. Only TCP connections get compressed. Local
62
73
  transports are unaffected even on the same socket. Dictionary lifetime
63
74
  matches connection lifetime naturally (new connection = new wrapper =
64
- re-ship dictionary). No negotiation is needed — both peers use
75
+ re-ship dictionary). No negotiation is needed; both peers use
65
76
  `zstd+tcp://`. The codec is socket-wide (shared across connections), so
66
77
  fan-out patterns compress once and reuse the result.
67
78
 
68
- ### 2.2 Why not negotiate
79
+ ### 3.2 Why not negotiate
69
80
 
70
- ZMTP 3.1 already supports unknown READY properties — an unaware peer
81
+ ZMTP 3.1 already supports unknown READY properties. An unaware peer
71
82
  silently ignores them. A negotiation-based design could fall back to
72
83
  plaintext when the peer does not understand compression. But this
73
84
  introduces complexity (profile matching, asymmetric per-direction state,
@@ -76,7 +87,7 @@ deployment decision, not a runtime discovery. Both peers are configured
76
87
  to use `zstd+tcp://` or they are not. The transport scheme approach
77
88
  eliminates the entire negotiation surface and its edge cases.
78
89
 
79
- ### 2.3 Why Zstandard
90
+ ### 3.3 Why Zstandard
80
91
 
81
92
  Zstandard at low levels matches LZ4 on encode latency, beats it on
82
93
  decompression speed and ratio at every realistic ZMQ payload size, and
@@ -85,9 +96,10 @@ particularly important for fan-out patterns (PUB/SUB, RADIO/DISH): the
85
96
  publisher pays one compress, every subscriber pays decompress, so
86
97
  per-subscriber CPU dominates the total budget.
87
98
 
88
- ## 3. Goals and Non-goals
89
99
 
90
- ### 3.1 Goals
100
+ ## 4. Goals and Non-goals
101
+
102
+ ### 4.1 Goals
91
103
 
92
104
  - Transparent to application code: send/receive operations see plaintext.
93
105
  - Per-part sender decision: opt out for short or incompressible parts.
@@ -98,50 +110,52 @@ per-subscriber CPU dominates the total budget.
98
110
  - No ZMTP-level negotiation, no new READY properties, no new command
99
111
  frames.
100
112
 
101
- ### 3.2 Non-goals
113
+ ### 4.2 Non-goals
102
114
 
103
115
  - New ZMTP mechanism, new socket type, new greeting, new frame flag bit.
104
116
  - Compression of the ZMTP greeting or command frames (READY, SUBSCRIBE,
105
117
  PING, PONG, ...).
106
- - Application to non-TCP transports (`inproc://` is zero-copy —
118
+ - Application to non-TCP transports (`inproc://` is zero-copy;
107
119
  compression is pure overhead; `ipc://` rarely benefits).
108
120
  - Replacing or weakening CurveZMQ or any other security mechanism.
109
- See Sec. 8.
121
+ See Sec. 9.
110
122
  - Streaming / context-takeover compression. Each part is decodable in
111
123
  isolation with no dependency on a previous part's LZ77 history.
112
124
 
113
- ## 4. Terminology
114
125
 
115
- | Term | Meaning |
116
- |---------------------|---------------------------------------------------------------------------|
117
- | Part | One ZMTP message frame body. A multipart message has multiple parts. |
118
- | Sentinel | The first 4 bytes of a post-handshake part on the wire (Sec. 5.1). |
119
- | Uncompressed part | A wire part whose sentinel is `00 00 00 00`. |
120
- | Compressed part | A wire part whose first 4 bytes are the Zstandard magic `28 B5 2F FD`. |
121
- | Dictionary part | A wire part whose first 4 bytes are `37 A4 30 EC` (Sec. 6). |
122
- | Dictionary message | A single-part ZMTP message consisting of exactly one dictionary part. |
126
+ ## 5. Terminology
127
+
128
+ | Term | Meaning |
129
+ |---------------------|----------------------------------------------------------------------|
130
+ | Part | One ZMTP message frame body. A multipart message has multiple parts. |
131
+ | Sentinel | The first 4 bytes of a post-handshake part on the wire (Sec. 6.1). |
132
+ | Uncompressed part | A wire part whose sentinel is `00 00 00 00`. |
133
+ | Compressed part | A wire part whose first 4 bytes are the Zstandard magic `28 B5 2F FD`. |
134
+ | Dictionary part | A wire part whose first 4 bytes are `37 A4 30 EC` (Sec. 7). |
135
+ | Dictionary message | A single-part ZMTP message consisting of exactly one dictionary part. |
123
136
 
124
- ## 5. Part Encoding
137
+
138
+ ## 6. Part Encoding
125
139
 
126
140
  After the ZMTP handshake completes, every message part on the wire is
127
141
  individually encoded. The ZMTP MORE flag is carried on the wire frame
128
- header as normal. Multipart messages are encoded part by part — each
142
+ header as normal. Multipart messages are encoded part by part; each
129
143
  part is independent.
130
144
 
131
- ### 5.1 Sentinel dispatch
145
+ ### 6.1 Sentinel dispatch
132
146
 
133
147
  The first 4 bytes of each wire part determine how it is decoded.
134
148
 
135
- | Sentinel (hex) | Meaning |
136
- |------------------|-----------------------------------------------------------------|
137
- | `00 00 00 00` | Uncompressed plaintext (Sec. 5.3) |
138
- | `28 B5 2F FD` | Zstandard compressed frame (Sec. 5.4) |
139
- | `37 A4 30 EC` | Dictionary shipment (Sec. 6) |
149
+ | Sentinel (hex) | Meaning |
150
+ |------------------|--------------------------------------|
151
+ | `00 00 00 00` | Uncompressed plaintext (Sec. 6.3) |
152
+ | `28 B5 2F FD` | Zstandard compressed frame (Sec. 6.4)|
153
+ | `37 A4 30 EC` | Dictionary shipment (Sec. 7) |
140
154
 
141
155
  All other 4-byte values are reserved. A receiver that encounters an
142
- unknown sentinel MUST drop the connection with an error.
156
+ unknown sentinel MUST close the connection.
143
157
 
144
- ### 5.2 Compression level
158
+ ### 6.2 Compression level
145
159
 
146
160
  The default compression level is **-3** (Zstandard fast strategy). At
147
161
  this level the encoder cost is in the low single-digit microseconds per
@@ -149,11 +163,11 @@ kilobyte, and the achieved ratio is within a few percent of level 3 once
149
163
  a dictionary is in play.
150
164
 
151
165
  The compression level is a sender choice and is not communicated on the
152
- wire — the receiver decodes any valid Zstandard frame regardless of the
166
+ wire. The receiver decodes any valid Zstandard frame regardless of the
153
167
  level used to encode it. Implementations SHOULD expose the level as a
154
168
  configurable parameter.
155
169
 
156
- ### 5.3 Uncompressed sentinel `00 00 00 00`
170
+ ### 6.3 Uncompressed sentinel `00 00 00 00`
157
171
 
158
172
  ```
159
173
  +------------------+-------------------+
@@ -169,7 +183,7 @@ without an extra flag bit in the ZMTP frame header.
169
183
  Four zero bytes cannot collide with a valid Zstandard frame magic or the
170
184
  dictionary sentinel, so no ambiguity arises.
171
185
 
172
- ### 5.4 Compressed Zstandard frame
186
+ ### 6.4 Compressed Zstandard frame
173
187
 
174
188
  ```
175
189
  +------------------+
@@ -178,15 +192,15 @@ dictionary sentinel, so no ambiguity arises.
178
192
  +------------------+
179
193
  ```
180
194
 
181
- The wire part IS the Zstandard frame — its first 4 bytes are the
195
+ The wire part IS the Zstandard frame. Its first 4 bytes are the
182
196
  standard Zstandard frame magic `28 B5 2F FD`. No additional framing is
183
197
  added.
184
198
 
185
199
  The sender MUST configure the encoder to write the `Frame_Content_Size`
186
- field in the Zstandard frame header (RFC 8878 §3.1.1.1.2). This field
187
- is required for the receiver's budget enforcement (Sec. 5.6).
200
+ field in the Zstandard frame header (RFC 8878 Sec. 3.1.1.1.2). This
201
+ field is required for the receiver's budget enforcement (Sec. 6.6).
188
202
 
189
- ### 5.5 Sender rules
203
+ ### 6.5 Sender rules
190
204
 
191
205
  For each outgoing message part, the sender proceeds as follows:
192
206
 
@@ -203,7 +217,7 @@ For each outgoing message part, the sender proceeds as follows:
203
217
 
204
218
  3. Otherwise, run the Zstandard encoder. The encoder MUST write the
205
219
  `Frame_Content_Size` field. If the compressed output's size is
206
- ≥ `plaintext_size - 4` (net saving ≤ 0 after accounting for the
220
+ >= `plaintext_size - 4` (net saving <= 0 after accounting for the
207
221
  4-byte sentinel of the uncompressed alternative), prepend
208
222
  `00 00 00 00` and emit the plaintext instead. Otherwise emit the
209
223
  Zstandard frame as-is.
@@ -213,41 +227,42 @@ For each outgoing message part, the sender proceeds as follows:
213
227
  MUST still prepend `00 00 00 00` to avoid sentinel ambiguity.
214
228
  Step 2 and step 3's fallback path already guarantee this.
215
229
 
216
- ### 5.6 Receiver rules
230
+ ### 6.6 Receiver rules
217
231
 
218
232
  For each incoming wire part, the receiver proceeds as follows:
219
233
 
220
234
  1. Read the first 4 bytes as the sentinel. If the part is shorter than
221
- 4 bytes, drop the connection with an error.
235
+ 4 bytes, close the connection.
222
236
 
223
237
  2. Sentinel `00 00 00 00`: the remaining `N - 4` bytes are plaintext.
224
238
  Return them.
225
239
 
226
240
  3. Sentinel `28 B5 2F FD`: the entire wire part is a Zstandard frame.
227
241
  - Read the `Frame_Content_Size` field from the Zstandard header. If
228
- the field is absent, drop the connection with an error.
242
+ the field is absent, close the connection.
229
243
  - If the connection enforces a maximum message size, add this part's
230
244
  declared content size to the running decompressed total for the
231
245
  current multipart message (parts chained by the ZMTP MORE flag).
232
- If the running total would exceed the maximum, drop the connection
233
- with an error without invoking the decoder.
246
+ If the running total would exceed the maximum, close the connection
247
+ without invoking the decoder.
234
248
  - Invoke the decoder in a bounded mode that aborts if it would write
235
249
  more bytes than `Frame_Content_Size` declared. On such an abort,
236
- drop the connection with an error.
250
+ close the connection.
237
251
  - Return the decompressed plaintext.
238
252
 
239
- 4. Sentinel `37 A4 30 EC`: dictionary shipment. See Sec. 6.
253
+ 4. Sentinel `37 A4 30 EC`: dictionary shipment. See Sec. 7.
240
254
 
241
- 5. Any other sentinel: drop the connection with an error.
255
+ 5. Any other sentinel: close the connection.
242
256
 
243
257
  The maximum message size always refers to the **decompressed** plaintext
244
258
  summed across all parts of a multipart message. A multipart message
245
259
  whose total wire length is small but whose total decompressed size
246
260
  exceeds the limit MUST be rejected before decoder invocation.
247
261
 
248
- ## 6. Dictionary Shipment
249
262
 
250
- ### 6.1 Dictionary message format
263
+ ## 7. Dictionary Shipment
264
+
265
+ ### 7.1 Dictionary message format
251
266
 
252
267
  A dictionary is shipped as a **single-part ZMTP message** (no MORE flag)
253
268
  whose body begins with the dictionary sentinel:
@@ -266,40 +281,40 @@ with the Zstandard frame magic and the uncompressed sentinel.
266
281
  The remaining `D` bytes are the raw dictionary as it should be passed
267
282
  to the Zstandard decoder's dictionary-load operation.
268
283
 
269
- ### 6.2 Constraints
284
+ ### 7.2 Constraints
270
285
 
271
286
  - A dictionary message MUST be a single-part ZMTP message (MORE flag
272
287
  not set on the frame header). A dictionary sentinel in a multipart
273
288
  message's non-final or non-only part is a protocol error.
274
289
 
275
- - A dictionary message MUST NOT exceed **64 KiB** total (sentinel +
290
+ - A dictionary message MUST NOT exceed **8 KiB** total (sentinel +
276
291
  dictionary bytes). A receiver that receives a dictionary message
277
- larger than 64 KiB MUST drop the connection with an error.
292
+ larger than 8 KiB MUST close the connection.
278
293
 
279
294
  - A sender MUST send at most **one** dictionary message per direction
280
295
  per connection. A receiver that receives a second dictionary message
281
- on the same connection MUST drop the connection with an error.
296
+ on the same connection MUST close the connection.
282
297
 
283
298
  - A dictionary message MUST be sent BEFORE any compressed part that
284
299
  references the dictionary. In practice this means the sender ships
285
300
  the dictionary before (or immediately after training triggers during)
286
301
  the first compressed write that would benefit from it.
287
302
 
288
- ### 6.3 Receiver handling
303
+ ### 7.3 Receiver handling
289
304
 
290
305
  When the receiver encounters a dictionary part:
291
306
 
292
- 1. Validate the constraints in Sec. 6.2.
307
+ 1. Validate the constraints in Sec. 7.2.
293
308
  2. Strip the 4-byte sentinel.
294
309
  3. Install the remaining bytes as the decompression dictionary for this
295
310
  connection.
296
- 4. Discard the message — it is not delivered to the application.
311
+ 4. Discard the message. It is not delivered to the application.
297
312
 
298
313
  If all parts of a ZMTP message are dictionary parts (which is always
299
314
  the case, since dictionary messages are single-part), the receiver
300
315
  loops to receive the next message.
301
316
 
302
- ### 6.4 Dictionary scope
317
+ ### 7.4 Dictionary scope
303
318
 
304
319
  The dictionary a sender ships applies to a single direction of a single
305
320
  connection. Each peer may independently ship its own dictionary for its
@@ -309,7 +324,7 @@ nothing (or uncompressed traffic) back.
309
324
 
310
325
  The sender's dictionary is typically socket-wide: trained once from
311
326
  early traffic across all connections and reused. But this is an
312
- implementation choice — the wire protocol carries no dictionary identity
327
+ implementation choice. The wire protocol carries no dictionary identity
313
328
  or scope metadata.
314
329
 
315
330
  An implementation MAY pool training samples and share the resulting
@@ -319,38 +334,48 @@ multiple `zstd+tcp://` endpoints: samples from one endpoint accelerate
319
334
  training for all of them, and newly opened connections benefit from a
320
335
  dictionary trained by their predecessors. Connections that were
321
336
  configured with an explicit out-of-band dictionary MUST NOT participate
322
- in shared training — they use their own dictionary independently.
337
+ in shared training; they use their own dictionary independently.
323
338
 
324
- ### 6.5 Automatic dictionary training
339
+ ### 7.5 Automatic dictionary training
325
340
 
326
- A sender MAY train a dictionary automatically from early traffic:
341
+ A sender MAY train a dictionary automatically from early traffic.
342
+ The training algorithm, trigger condition, and parameters are
343
+ implementation choices. The following wire-level constraints MUST be
344
+ respected:
327
345
 
328
- 1. Buffer plaintext samples from the first messages. Samples larger
329
- than **1024 bytes** SHOULD be skipped — dictionaries primarily
330
- benefit small frames.
331
- 2. When the buffer reaches **1000 samples** OR **100 KiB** of
332
- plaintext (whichever comes first), train a Zstandard dictionary from
333
- the buffered samples and discard the buffer.
334
- 3. The recommended dictionary capacity (training target size) is
335
- **8 KiB**.
336
- 4. Ship the trained dictionary via a dictionary message (Sec. 6.1) on
337
- every connection, before any compressed part that uses it.
338
- 5. Switch to dictionary-bound compression for all subsequent parts.
346
+ - A trained dictionary MUST NOT exceed the maximum dictionary size
347
+ (Sec. 7.2: 8 KiB).
348
+ - A sender MUST ship at most one dictionary per direction per
349
+ connection (Sec. 7.2).
350
+ - If training fails (the sample set was too small or too uniform),
351
+ the sender MUST stay in no-dictionary mode for the rest of the
352
+ socket's lifetime. It MUST NOT retry training.
339
353
 
340
- If training fails (the sample set was too small or too uniform), the
341
- sender MUST stay in no-dictionary mode for the rest of the socket's
342
- lifetime. It MUST NOT retry training.
354
+ The following parameters are RECOMMENDED for implementations that
355
+ support auto-training:
343
356
 
344
- ### 6.6 Dictionary ID
357
+ | Parameter | Recommended value |
358
+ |------------------------|------------------------------------------|
359
+ | Dictionary capacity | 2 KiB |
360
+ | Training trigger | 1000 samples OR 100 KiB (first reached) |
361
+ | Max sample length | 2048 bytes |
362
+ | Training algorithm | FastCOVER |
363
+
364
+ Whether auto-training is enabled by default is an implementation
365
+ choice. Applications MAY disable it or supply an out-of-band
366
+ dictionary instead.
367
+
368
+ ### 7.6 Dictionary ID
345
369
 
346
370
  Auto-trained dictionaries SHOULD be patched with a random dictionary ID
347
371
  in the Zstandard user range (32768 to 2^31 - 1) to avoid collisions
348
372
  with Zstandard's built-in dictionary IDs. Out-of-band dictionaries
349
373
  retain whatever dictionary ID they were created with.
350
374
 
351
- ## 7. ZMTP Interaction
352
375
 
353
- ### 7.1 Greeting and handshake
376
+ ## 8. ZMTP Interaction
377
+
378
+ ### 8.1 Greeting and handshake
354
379
 
355
380
  The ZMTP greeting and security mechanism handshake proceed over raw TCP
356
381
  exactly as specified by RFC 37. `zstd+tcp://` does not modify the
@@ -358,28 +383,29 @@ greeting, mechanism, READY properties, or any command frames. The
358
383
  compression layer activates only after the handshake is complete and the
359
384
  connection is ready for message traffic.
360
385
 
361
- ### 7.2 Command frames
386
+ ### 8.2 Command frames
362
387
 
363
388
  ZMTP command frames (READY, SUBSCRIBE, CANCEL, JOIN, LEAVE, PING,
364
389
  PONG) are never compressed. They are sent and received as standard ZMTP
365
390
  command frames. Only message frames (the COMMAND bit not set in the
366
391
  frame header) are subject to sentinel-dispatched encoding.
367
392
 
368
- ### 7.3 Socket type compatibility
393
+ ### 8.3 Socket type compatibility
369
394
 
370
395
  `zstd+tcp://` is compatible with all ZMTP socket types. The socket type
371
396
  negotiation in the READY handshake is unaffected.
372
397
 
373
- ### 7.4 Peer requirement
398
+ ### 8.4 Peer requirement
374
399
 
375
400
  Both peers of a connection MUST use `zstd+tcp://`. There is no
376
401
  fallback to plaintext TCP and no negotiation. A `zstd+tcp://` peer
377
402
  connecting to a plain `tcp://` peer (or vice versa) will see garbled
378
403
  data or sentinel errors and the connection will fail.
379
404
 
380
- ## 8. Security Considerations
381
405
 
382
- ### 8.1 Compression combined with encryption (CRIME / BREACH)
406
+ ## 9. Security Considerations
407
+
408
+ ### 9.1 Compression combined with encryption (CRIME / BREACH)
383
409
 
384
410
  Combining length-revealing compression with a secure channel that
385
411
  carries attacker-influenced plaintext enables CRIME- and BREACH-style
@@ -392,30 +418,30 @@ encrypted tunnel when the plaintext contains attacker-controlled
392
418
  content. Deployments that accept this risk MUST do so with explicit
393
419
  opt-in.
394
420
 
395
- ### 8.2 Length side-channel
421
+ ### 9.2 Length side-channel
396
422
 
397
423
  Compression makes the wire length of a part depend on its content. An
398
424
  on-path observer can learn something about the plaintext from the
399
425
  compressed length alone. Deployments that care about traffic analysis
400
426
  MUST NOT rely on `zstd+tcp://` to hide payload shape.
401
427
 
402
- ### 8.3 Dictionary contents
428
+ ### 9.3 Dictionary contents
403
429
 
404
430
  When auto-training is enabled, the receiver loads dictionary bytes
405
431
  chosen by the peer. The Zstandard reference dictionary loader is
406
432
  hardened against malformed inputs, but implementations MUST enforce the
407
- 64 KiB cap on dictionary messages (Sec. 6.2) and SHOULD NOT cache
433
+ 8 KiB cap on dictionary messages (Sec. 7.2) and SHOULD NOT cache
408
434
  received dictionaries across connections.
409
435
 
410
- ### 8.4 Decompression bombs
436
+ ### 9.4 Decompression bombs
411
437
 
412
438
  A small compressed frame can decompress to many megabytes of plaintext.
413
- The receiver rules in Sec. 5.6 mitigate this:
439
+ The receiver rules in Sec. 6.6 mitigate this:
414
440
 
415
441
  1. Every compressed part MUST carry `Frame_Content_Size`. The receiver
416
442
  checks the declared total against the maximum message size before
417
443
  invoking the decoder, so a bomb is rejected on its header alone.
418
- 2. The decoder is invoked in bounded mode — it aborts if it would write
444
+ 2. The decoder is invoked in bounded mode. It aborts if it would write
419
445
  more bytes than declared. A peer that lies in the header cannot
420
446
  expand a part past its declared size.
421
447
 
@@ -423,30 +449,30 @@ Implementations SHOULD set a conservative maximum message size on
423
449
  `zstd+tcp://` connections even if they would otherwise leave it
424
450
  unbounded.
425
451
 
426
- ## 9. Constants
427
452
 
428
- ```
429
- SENTINEL_UNCOMPRESSED = 00 00 00 00 (4 bytes)
430
- SENTINEL_ZSTD_FRAME = 28 B5 2F FD (4 bytes, Zstandard frame magic)
431
- SENTINEL_ZSTD_DICT = 37 A4 30 EC (4 bytes)
432
-
433
- DEFAULT_LEVEL = -3
453
+ ## 10. Constants
434
454
 
435
- MIN_COMPRESS_NO_DICT = 512 bytes
436
- MIN_COMPRESS_WITH_DICT = 64 bytes
455
+ | Constant | Value |
456
+ |-------------------------|-----------------------------------------------|
457
+ | Uncompressed sentinel | `00 00 00 00` |
458
+ | Zstd frame sentinel | `28 B5 2F FD` (Zstandard frame magic) |
459
+ | Dictionary sentinel | `37 A4 30 EC` |
460
+ | Default level | -3 |
461
+ | Min compress, no dict | 512 bytes |
462
+ | Min compress, with dict | 64 bytes |
463
+ | Max dictionary size | 8 KiB |
464
+ | Train max samples | 1000 |
465
+ | Train max bytes | 100 KiB |
466
+ | Train max sample length | 2048 bytes |
467
+ | Dictionary capacity | 2 KiB |
437
468
 
438
- MAX_DICT_SIZE = 64 KiB
439
-
440
- TRAIN_MAX_SAMPLES = 1000
441
- TRAIN_MAX_BYTES = 100 KiB
442
- TRAIN_MAX_SAMPLE_LEN = 1024 bytes
443
- DICT_CAPACITY = 8 KiB
444
- ```
445
469
 
446
- ## 10. References
470
+ ## 11. References
447
471
 
448
- - [RFC 37 / ZMTP 3.1](https://rfc.zeromq.org/spec/37/) — underlying wire protocol
449
- - [RFC 8878 — Zstandard Compression Data Format](https://datatracker.ietf.org/doc/html/rfc8878)
472
+ - [RFC 37/ZMTP 3.1](https://rfc.zeromq.org/spec/37/)
473
+ - [RFC 2119](https://datatracker.ietf.org/doc/html/rfc2119)
474
+ - [RFC 8878: Zstandard Compression Data Format](https://datatracker.ietf.org/doc/html/rfc8878)
450
475
  - [Zstandard dictionary builder](https://github.com/facebook/zstd/blob/dev/lib/dictBuilder/zdict.h)
451
- - [CRIME attack](https://en.wikipedia.org/wiki/CRIME) — compression side-channel on TLS
452
- - [BREACH attack](https://en.wikipedia.org/wiki/BREACH) — HTTP-layer variant
476
+ - [CRIME attack](https://en.wikipedia.org/wiki/CRIME)
477
+ - [BREACH attack](https://en.wikipedia.org/wiki/BREACH)
478
+ - [`lz4+tcp://` RFC](../omq-lz4/RFC.md)
data/lib/omq/transport/zstd_tcp/codec.rb CHANGED
@@ -4,11 +4,11 @@ module OMQ
4
4
  module Transport
5
5
  module ZstdTcp
6
6
  class Codec
7
- MAX_DICT_SIZE = 64 * 1024
8
- DICT_CAPACITY = 8 * 1024
7
+ MAX_DICT_SIZE = 8 * 1024
8
+ DICT_CAPACITY = 2 * 1024
9
9
  TRAIN_MAX_SAMPLES = 1000
10
10
  TRAIN_MAX_BYTES = 100 * 1024
11
- TRAIN_MAX_SAMPLE_LEN = 1024
11
+ TRAIN_MAX_SAMPLE_LEN = 2048
12
12
  MIN_COMPRESS_NO_DICT = 512
13
13
  MIN_COMPRESS_WITH_DICT = 64
14
14
 
@@ -28,10 +28,10 @@ module OMQ
28
28
 
29
29
  # Start with a no-dict FrameCodec. Once a dict is configured
30
30
  # (either via the `dict:` kwarg or via auto-training), this is
31
- # replaced with a fresh dict-bound FrameCodec — per rzstd 0.4,
31
+ # replaced with a fresh dict-bound FrameCodec. In Zrip, the
32
32
  # dict is a permanent property of the codec, so swapping dicts
33
33
  # means constructing a new one.
34
- @send_codec = RZstd::FrameCodec.new(level: @level)
34
+ @send_codec = Zrip::FrameCodec.new(level: @level)
35
35
  @send_dict_bytes = nil
36
36
 
37
37
  @training = dict.nil?
@@ -102,18 +102,16 @@ module OMQ
102
102
  return unless @train_samples.size >= TRAIN_MAX_SAMPLES ||
103
103
  @train_bytes >= TRAIN_MAX_BYTES
104
104
 
105
- begin
106
- trained = RZstd::Dictionary.train(@train_samples, capacity: DICT_CAPACITY)
107
- rescue RuntimeError
108
- @training = false
109
- @train_samples = nil
110
- return
111
- end
105
+ trainer = Zrip::DictTrainer.new(DICT_CAPACITY)
106
+ @train_samples.each { |s| trainer.add_sample(s) }
107
+ trained_bytes = trainer.train
112
108
 
113
109
  @training = false
114
110
  @train_samples = nil
115
111
 
116
- patched = patch_auto_dict_id(trained.bytes)
112
+ return if trained_bytes.empty?
113
+
114
+ patched = patch_auto_dict_id(trained_bytes)
117
115
  install_send_dict(patched)
118
116
  end
119
117
 
@@ -136,9 +134,9 @@ module OMQ
136
134
  end
137
135
 
138
136
  # Replace the no-dict send codec with a fresh dict-bound one;
139
- # the old codec is GC'd. rzstd 0.4 treats dict as a permanent
140
- # codec property, so install-time is always a fresh build.
141
- @send_codec = RZstd::FrameCodec.new(dict: bytes, level: @level)
137
+ # the old codec is GC'd. Zrip treats dict as a permanent codec
138
+ # property, so install-time is always a fresh build.
139
+ @send_codec = Zrip::FrameCodec.new(dict: bytes, level: @level)
142
140
  @send_dict_bytes = bytes
143
141
  end
144
142
 
data/lib/omq/transport/zstd_tcp/connection.rb CHANGED
@@ -15,12 +15,11 @@ module OMQ
15
15
  super(conn)
16
16
  @codec = codec
17
17
  @dict_shipped = false
18
- # rzstd 0.4: FrameCodec is the decoder. Starts no-dict; when a
19
- # dict shipment arrives on this direction, we build a fresh
20
- # dict-bound FrameCodec and replace this one. Level is a
21
- # compression parameter only — not consulted on decompress —
22
- # so we don't carry the send-side level through to recv.
23
- @recv_codec = RZstd::FrameCodec.new
18
+ # FrameCodec is the decoder. Starts no-dict; when a dict
19
+ # shipment arrives on this direction, we build a fresh
20
+ # dict-bound FrameCodec and replace this one.
21
+ @recv_codec = Zrip::FrameCodec.new
22
+ @recv_no_dict_codec = @recv_codec
24
23
  @recv_dict_bytes = nil
25
24
  @last_wire_size_in = nil
26
25
  end
@@ -128,16 +127,24 @@ module OMQ
128
127
 
129
128
  decompress_opts = budget ? { max_output_size: budget } : {}
130
129
 
131
- @recv_codec.decompress(wire, **decompress_opts)
132
- rescue RZstd::DecompressError => e
130
+ codec = frame_has_dict_id?(wire) ? @recv_codec : @recv_no_dict_codec
131
+ codec.decompress(wire, **decompress_opts)
132
+ rescue Zrip::DecompressError => e
133
133
  raise ProtocolError, "decompression failed: #{e.message}"
134
- rescue RZstd::MissingContentSizeError => e
134
+ rescue Zrip::MissingContentSizeError => e
135
135
  raise ProtocolError, "Zstd frame missing Frame_Content_Size (#{e.message})"
136
- rescue RZstd::OutputSizeLimitError => e
136
+ rescue Zrip::OutputSizeLimitError => e
137
137
  raise ProtocolError, "declared FCS exceeds limit (#{e.message})"
138
138
  end
139
139
 
140
140
 
141
+ def frame_has_dict_id?(wire)
142
+ return false if wire.bytesize < 5
143
+
144
+ (wire.getbyte(4) & 0x03) != 0
145
+ end
146
+
147
+
141
148
  def install_recv_dict(wire)
142
149
  if wire.bytesize < 8
143
150
  raise ProtocolError, "dict frame too short"
@@ -147,10 +154,7 @@ module OMQ
147
154
  raise ProtocolError, "dict exceeds #{Codec::MAX_DICT_SIZE} bytes"
148
155
  end
149
156
 
150
- # Replace the no-dict recv codec with a fresh dict-bound one;
151
- # the old codec is GC'd (rzstd 0.4: dict is a permanent codec
152
- # property).
153
- @recv_codec = RZstd::FrameCodec.new(dict: wire.b)
157
+ @recv_codec = Zrip::FrameCodec.new(dict: wire.b)
154
158
  @recv_dict_bytes = wire.b
155
159
  end
156
160
 
data/lib/omq/transport/zstd_tcp.rb CHANGED
@@ -9,7 +9,7 @@
9
9
  # push.connect("zstd+tcp://127.0.0.1:5555", level: -3)
10
10
 
11
11
  require "omq"
12
- require "rzstd"
12
+ require "zrip"
13
13
 
14
14
  require_relative "zstd_tcp/codec"
15
15
  require_relative "zstd_tcp/connection"
data/lib/omq/zstd/version.rb CHANGED
@@ -2,6 +2,6 @@
2
2
 
3
3
  module OMQ
4
4
  module Zstd
5
- VERSION = "0.4.1"
5
+ VERSION = "0.4.3"
6
6
  end
7
7
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: omq-zstd
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.1
4
+ version: 0.4.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Patrik Wenger
@@ -15,30 +15,31 @@ dependencies:
15
15
  requirements:
16
16
  - - "~>"
17
17
  - !ruby/object:Gem::Version
18
- version: '0.23'
18
+ version: '0.28'
19
19
  type: :runtime
20
20
  prerelease: false
21
21
  version_requirements: !ruby/object:Gem::Requirement
22
22
  requirements:
23
23
  - - "~>"
24
24
  - !ruby/object:Gem::Version
25
- version: '0.23'
25
+ version: '0.28'
26
26
  - !ruby/object:Gem::Dependency
27
- name: rzstd
27
+ name: zrip
28
28
  requirement: !ruby/object:Gem::Requirement
29
29
  requirements:
30
30
  - - "~>"
31
31
  - !ruby/object:Gem::Version
32
- version: '0.4'
32
+ version: 0.1.1
33
33
  type: :runtime
34
34
  prerelease: false
35
35
  version_requirements: !ruby/object:Gem::Requirement
36
36
  requirements:
37
37
  - - "~>"
38
38
  - !ruby/object:Gem::Version
39
- version: '0.4'
40
- description: Adds zstd+tcp:// endpoint support to OMQ with per-frame Zstd compression,
41
- bounded decompression, in-band dictionary shipping, and sender-side dictionary training.
39
+ version: 0.1.1
40
+ description: Experimental zstd+tcp:// endpoint support for OMQ with per-frame Zstd
41
+ compression, bounded decompression, in-band dictionary shipping, and sender-side
42
+ dictionary training.
42
43
  email:
43
44
  - paddor@gmail.com
44
45
  executables: []
@@ -55,7 +56,7 @@ files:
55
56
  - lib/omq/transport/zstd_tcp/transport.rb
56
57
  - lib/omq/zstd.rb
57
58
  - lib/omq/zstd/version.rb
58
- homepage: https://github.com/paddor/omq-zstd
59
+ homepage: https://github.com/zeromq/omq.rb/tree/main/gems/omq-zstd
59
60
  licenses:
60
61
  - ISC
61
62
  metadata: {}
@@ -66,14 +67,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
66
67
  requirements:
67
68
  - - ">="
68
69
  - !ruby/object:Gem::Version
69
- version: '3.3'
70
+ version: '4.0'
70
71
  required_rubygems_version: !ruby/object:Gem::Requirement
71
72
  requirements:
72
73
  - - ">="
73
74
  - !ruby/object:Gem::Version
74
75
  version: '0'
75
76
  requirements: []
76
- rubygems_version: 4.0.6
77
+ rubygems_version: 4.0.16
77
78
  specification_version: 4
78
- summary: Zstd+TCP transport for OMQ
79
+ summary: Experimental Zstd+TCP transport for OMQ
79
80
  test_files: []