omq-zstd 0.4.2 → 0.4.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1fcea64306ec7603681d7b3350e7cdf2d22d43487be062e79cc91f51f64aacfb
-  data.tar.gz: ac5d8d88deb3f90ccf97be757d384612274f4d742b042b79bc949a47ea7d0d3c
+  metadata.gz: 22b1f25991e512b445fe5a7202d2671048159b494a7ac5418d1b121f696940ab
+  data.tar.gz: 941b1c5f85ec8dd1a9816fbfac241f882be1f2e9cd80696b1cf894465e90d884
 SHA512:
-  metadata.gz: 8e5a04f9ffdb46d130af145fa7025c544cc1709e7711cd92180b0408923667e61d70098efc4679c7b0d2617a3acde40b91c9ecaa1b5abb9e56cadc4a34cd8294
-  data.tar.gz: 25ab87f733a74257a9b2d772d96a76be134b60804ce2796fa0a95710334eb0552f973fab4a7b04442f6d9cce8091da008c5ed17e6868e925b942fdc4dfefd0f6
+  metadata.gz: 6525d971daf7707cb46113848b2a70c6e29775f9d7a87c6e9dcbd13cd481eebdd7c6a75c919d629b281e491c3263ec56b2e149bc573e4e0f24df32f042247dab
+  data.tar.gz: 99d1a5e3d2ce007036a0712fc7912331bbda35437eca0976d55eb58b8d0eedb52fd5a285c9c025fa658c75231ff4e538fb0f1dcd71d0d38a6d67505a72c4ae1e

data/DESIGN.md

@@ -67,7 +67,7 @@ One Codec per Dialer or Listener. Shared across all connections of
 that endpoint. Owns:
 
 - **Compression**: `#compress_parts(parts)` with identity cache
-- **Training**: sample collection, `RZstd::Dictionary.train`, dict ID patching
+- **Training**: sample collection, `Zrip::DictTrainer`, dict ID patching
 - **Send dict**: `#send_dict_bytes` — the trained or user-supplied dict bytes
 
 ### ZstdConnection (per-connection)
@@ -124,7 +124,7 @@ end
 - **Sample size cap**: frames > 1024 bytes are skipped (dictionaries primarily benefit small frames)
 - **Dict capacity**: 8 KiB (conservative; Zstd recommends ~100:1 sample-to-dict ratio)
 - **Dict ID patching**: auto-trained dicts get a random ID in the user range (32768..2^31-1) to avoid collisions with Zstd's built-in dict IDs
-- **Training failure**: if `RZstd::Dictionary.train` raises, training is disabled permanently for the socket. No retry.
+- **Training failure**: if `Zrip::DictTrainer#train` raises, training is disabled permanently for the socket. No retry.
 
 ## Frame dispatch
 

data/README.md

@@ -2,13 +2,19 @@
 
 [![Gem Version](https://img.shields.io/gem/v/omq-zstd?color=e9573f)](https://rubygems.org/gems/omq-zstd)
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
-[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%204.0-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+
+Experimental Zstandard-compressed TCP transport for
+[OMQ](https://github.com/zeromq/omq.rb).
+
+Use [`omq-lz4`](../omq-lz4) for CPU- or memory-sensitive compressed TCP
+work. This gem remains for research, comparison, OMQ.rs parity, and cases
+where zstd ratio matters more than transport simplicity.
 
-Zstandard-compressed TCP transport for [OMQ](https://github.com/paddor/omq).
 Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
 compressed per-part with [Zstandard](https://github.com/facebook/zstd).
 Compression is intrinsic to the transport: no negotiation, no socket option,
-no payload changes. The ZMTP handshake runs over plain TCP; only
+no payload changes. The ZMTP handshake runs over plain TCP. Only
 post-handshake message parts are compressed.
 
 See [RFC.md](RFC.md) for the wire-format specification and
@@ -73,11 +79,17 @@ The sender ships the dictionary to the receiver in-band as a one-shot
 single-part message prefixed with the dictionary sentinel
 (`37 A4 30 EC`), so the receiver does not need a copy on disk.
 
-**Auto-trained dictionary** (zero config, the default when no `dict:` is
-passed): the sender collects up to 1000 samples or 100 KiB (whichever hits
-first), skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary,
-ships it inline, and switches to dictionary mode. Until then, payloads are
-compressed without a dictionary or sent plaintext when below the threshold.
+**Auto-trained dictionary** (opt-in):
+
+```ruby
+push.connect("zstd+tcp://127.0.0.1:5555", auto_dict: true)
+```
+
+The sender collects up to 1000 samples or 100 KiB (whichever hits first),
+skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary, ships
+it inline, and switches to dictionary mode. Until then, payloads are compressed
+without a dictionary or sent plaintext when below the threshold. Automatic
+dictionary training is off by default.
 
 ### Compression thresholds
 
@@ -145,6 +157,7 @@ transport layer. They are not delivered to the application.
 | Train max bytes | 100 KiB |
 | Train max sample length | 2048 B |
 | Dictionary capacity | 2 KiB |
+| Auto dictionary training | Off by default |
 
 ## When to use it
 
@@ -170,7 +183,7 @@ It is **not** worth it for:
 `require "omq/zstd"` registers the `zstd+tcp` scheme on
 `OMQ::Engine.transports`. A `zstd+tcp` socket builds a per-engine
 `Codec` (one Zstd dictionary instance shared across all the socket's
-connections; fan-out compresses each part exactly once). Each accepted
+connections. Fan-out compresses each part exactly once). Each accepted
 or dialed TCP connection is wrapped in `ZstdConnection`, a
 `SimpleDelegator` over the underlying ZMTP connection that intercepts
 `#send_message` / `#write_message` / `#receive_message`. Message parts

data/RFC.md

@@ -265,21 +265,26 @@ exceeds the limit MUST be rejected before decoder invocation.
 ### 7.1 Dictionary message format
 
 A dictionary is shipped as a **single-part ZMTP message** (no MORE flag)
-whose body begins with the dictionary sentinel:
+whose body is exactly a Zstandard dictionary blob in ZDICT format. ZDICT
+blobs begin with the dictionary magic `37 A4 30 EC`; this transport uses
+those first 4 bytes as the dictionary sentinel:
 
 ```
 +------------------+------------------------+
-| 37 A4 30 EC      | dictionary bytes       |
-| (4 bytes)        | (D bytes)              |
+| 37 A4 30 EC      | rest of ZDICT blob     |
+| (4 bytes)        | (D - 4 bytes)          |
 +------------------+------------------------+
 ```
 
-The sentinel `37 A4 30 EC` is specific to this specification and has no
-relationship to Zstandard's internals. It was chosen to avoid collision
-with the Zstandard frame magic and the uncompressed sentinel.
+The sentinel `37 A4 30 EC` is the standard Zstandard dictionary magic
+(`DICT_MAGIC`, little-endian). It is intentionally reused as this
+transport's dictionary discriminator because a ZDICT blob already carries
+it and it does not collide with the Zstandard frame magic or the
+uncompressed sentinel.
 
-The remaining `D` bytes are the raw dictionary as it should be passed
-to the Zstandard decoder's dictionary-load operation.
+The complete `D` bytes, including the leading `37 A4 30 EC` magic, are
+the raw dictionary as it should be passed to the Zstandard decoder's
+dictionary-load operation.
 
 ### 7.2 Constraints
 
@@ -287,9 +292,10 @@ to the Zstandard decoder's dictionary-load operation.
   not set on the frame header). A dictionary sentinel in a multipart
   message's non-final or non-only part is a protocol error.
 
-- A dictionary message MUST NOT exceed **8 KiB** total (sentinel +
-  dictionary bytes). A receiver that receives a dictionary message
-  larger than 8 KiB MUST close the connection.
+- A dictionary message MUST NOT exceed **8 KiB** total. Since the
+  message body is exactly the ZDICT blob, this is also the maximum
+  dictionary size. A receiver that receives a dictionary message larger
+  than 8 KiB MUST close the connection.
 
 - A sender MUST send at most **one** dictionary message per direction
   per connection. A receiver that receives a second dictionary message
@@ -305,9 +311,9 @@ to the Zstandard decoder's dictionary-load operation.
 When the receiver encounters a dictionary part:
 
 1. Validate the constraints in Sec. 7.2.
-2. Strip the 4-byte sentinel.
-3. Install the remaining bytes as the decompression dictionary for this
-   connection.
+2. Install the complete message body as the decompression dictionary for
+   this connection. The leading `37 A4 30 EC` bytes are part of the ZDICT
+   blob and MUST NOT be stripped.
 4. Discard the message. It is not delivered to the application.
 
 If all parts of a ZMTP message are dictionary parts (which is always
@@ -338,23 +344,31 @@ in shared training; they use their own dictionary independently.
 
 ### 7.5 Automatic dictionary training
 
-A sender MAY train a dictionary automatically from early traffic:
-
-1. Buffer plaintext samples from the first messages. Samples larger
-   than **2048 bytes** SHOULD be skipped; dictionaries primarily
-   benefit small frames.
-2. When the buffer reaches **1000 samples** OR **100 KiB** of
-   plaintext (whichever comes first), train a Zstandard dictionary from
-   the buffered samples and discard the buffer.
-3. The recommended dictionary capacity (training target size) is
-   **2 KiB**.
-4. Ship the trained dictionary via a dictionary message (Sec. 7.1) on
-   every connection, before any compressed part that uses it.
-5. Switch to dictionary-bound compression for all subsequent parts.
-
-If training fails (the sample set was too small or too uniform), the
-sender MUST stay in no-dictionary mode for the rest of the socket's
-lifetime. It MUST NOT retry training.
+A sender MAY train a dictionary automatically from early traffic.
+The training algorithm, trigger condition, and parameters are
+implementation choices. The following wire-level constraints MUST be
+respected:
+
+- A trained dictionary MUST NOT exceed the maximum dictionary size
+  (Sec. 7.2: 8 KiB).
+- A sender MUST ship at most one dictionary per direction per
+  connection (Sec. 7.2).
+- If training fails (the sample set was too small or too uniform),
+  the sender MUST stay in no-dictionary mode for the rest of the
+  socket's lifetime. It MUST NOT retry training.
+
+The following parameters are RECOMMENDED for implementations that
+support auto-training:
+
+| Parameter              | Recommended value                        |
+|------------------------|------------------------------------------|
+| Dictionary capacity    | 2 KiB                                    |
+| Training trigger       | 1000 samples OR 100 KiB (first reached)  |
+| Max sample length      | 2048 bytes                               |
+| Training algorithm     | FastCOVER                                |
+
+Auto-training SHOULD be disabled by default. Applications MAY enable it
+explicitly or supply an out-of-band dictionary instead.
 
 ### 7.6 Dictionary ID
 

data/lib/omq/transport/zstd_tcp/codec.rb

@@ -22,19 +22,24 @@ module OMQ
         attr_reader :send_dict_bytes, :max_message_size
 
 
-        def initialize(level:, dict: nil, max_message_size: nil)
+        def initialize(level:, dict: nil, max_message_size: nil, auto_dict: nil)
           @level            = level
           @max_message_size = max_message_size
+          @dict_capacity    = if auto_dict.is_a?(Hash) && auto_dict[:capacity]
+                                auto_dict[:capacity]
+                              else
+                                DICT_CAPACITY
+                              end
 
           # Start with a no-dict FrameCodec. Once a dict is configured
           # (either via the `dict:` kwarg or via auto-training), this is
-          # replaced with a fresh dict-bound FrameCodec — per rzstd 0.4,
+          # replaced with a fresh dict-bound FrameCodec. In Zrip, the
           # dict is a permanent property of the codec, so swapping dicts
           # means constructing a new one.
-          @send_codec      = RZstd::FrameCodec.new(level: @level)
+          @send_codec      = Zrip::FrameCodec.new(level: @level)
           @send_dict_bytes = nil
 
-          @training      = dict.nil?
+          @training      = !!auto_dict && dict.nil?
           @train_samples = []
           @train_bytes   = 0
 
@@ -102,18 +107,16 @@ module OMQ
           return unless @train_samples.size >= TRAIN_MAX_SAMPLES ||
                         @train_bytes >= TRAIN_MAX_BYTES
 
-          begin
-            trained = RZstd::Dictionary.train(@train_samples, capacity: DICT_CAPACITY)
-          rescue RuntimeError
-            @training = false
-            @train_samples = nil
-            return
-          end
+          trainer = Zrip::DictTrainer.new(@dict_capacity)
+          @train_samples.each { |s| trainer.add_sample(s) }
+          trained_bytes = trainer.train
 
           @training = false
           @train_samples = nil
 
-          patched = patch_auto_dict_id(trained.bytes)
+          return if trained_bytes.empty?
+
+          patched = patch_auto_dict_id(trained_bytes)
           install_send_dict(patched)
         end
 
@@ -136,9 +139,9 @@ module OMQ
           end
 
           # Replace the no-dict send codec with a fresh dict-bound one;
-          # the old codec is GC'd. rzstd 0.4 treats dict as a permanent
-          # codec property, so install-time is always a fresh build.
-          @send_codec      = RZstd::FrameCodec.new(dict: bytes, level: @level)
+          # the old codec is GC'd. Zrip treats dict as a permanent codec
+          # property, so install-time is always a fresh build.
+          @send_codec      = Zrip::FrameCodec.new(dict: bytes, level: @level)
           @send_dict_bytes = bytes
         end
 

data/lib/omq/transport/zstd_tcp/connection.rb

@@ -15,12 +15,11 @@ module OMQ
           super(conn)
           @codec              = codec
           @dict_shipped       = false
-          # rzstd 0.4: FrameCodec is the decoder. Starts no-dict; when a
-          # dict shipment arrives on this direction, we build a fresh
-          # dict-bound FrameCodec and replace this one. Level is a
-          # compression parameter only — not consulted on decompress —
-          # so we don't carry the send-side level through to recv.
-          @recv_codec         = RZstd::FrameCodec.new
+          # FrameCodec is the decoder. Starts no-dict; when a dict
+          # shipment arrives on this direction, we build a fresh
+          # dict-bound FrameCodec and replace this one.
+          @recv_codec         = Zrip::FrameCodec.new
+          @recv_no_dict_codec = @recv_codec
           @recv_dict_bytes    = nil
           @last_wire_size_in  = nil
         end
@@ -128,16 +127,24 @@ module OMQ
 
           decompress_opts = budget ? { max_output_size: budget } : {}
 
-          @recv_codec.decompress(wire, **decompress_opts)
-        rescue RZstd::DecompressError => e
+          codec = frame_has_dict_id?(wire) ? @recv_codec : @recv_no_dict_codec
+          codec.decompress(wire, **decompress_opts)
+        rescue Zrip::DecompressError => e
           raise ProtocolError, "decompression failed: #{e.message}"
-        rescue RZstd::MissingContentSizeError => e
+        rescue Zrip::MissingContentSizeError => e
           raise ProtocolError, "Zstd frame missing Frame_Content_Size (#{e.message})"
-        rescue RZstd::OutputSizeLimitError => e
+        rescue Zrip::OutputSizeLimitError => e
           raise ProtocolError, "declared FCS exceeds limit (#{e.message})"
         end
 
 
+        def frame_has_dict_id?(wire)
+          return false if wire.bytesize < 5
+
+          (wire.getbyte(4) & 0x03) != 0
+        end
+
+
         def install_recv_dict(wire)
           if wire.bytesize < 8
             raise ProtocolError, "dict frame too short"
@@ -147,10 +154,7 @@ module OMQ
             raise ProtocolError, "dict exceeds #{Codec::MAX_DICT_SIZE} bytes"
           end
 
-          # Replace the no-dict recv codec with a fresh dict-bound one;
-          # the old codec is GC'd (rzstd 0.4: dict is a permanent codec
-          # property).
-          @recv_codec      = RZstd::FrameCodec.new(dict: wire.b)
+          @recv_codec      = Zrip::FrameCodec.new(dict: wire.b)
           @recv_dict_bytes = wire.b
         end
 

data/lib/omq/transport/zstd_tcp/transport.rb

@@ -24,10 +24,18 @@ module OMQ
         # @param engine [Engine]
         # @param level [Integer] Zstd compression level
         # @param dict [String, nil] user-supplied dictionary bytes
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. Pass `true` for defaults or `{ capacity: N }`.
         # @return [Listener]
         #
-        def listener(endpoint, engine, level: -3, dict: nil, **)
-          codec = codec_for(engine, level: level, dict: dict)
+        def listener(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_auto_dict!(auto_dict, dict)
+          codec = codec_for(
+            engine,
+            level: level,
+            dict: dict,
+            auto_dict: normalize_auto_dict(auto_dict),
+          )
 
           host, port = parse_endpoint(endpoint)
           host       = normalize_bind_host(host)
@@ -52,10 +60,18 @@ module OMQ
         # @param engine [Engine]
         # @param level [Integer] Zstd compression level
         # @param dict [String, nil] user-supplied dictionary bytes
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. Pass `true` for defaults or `{ capacity: N }`.
         # @return [Dialer]
         #
-        def dialer(endpoint, engine, level: -3, dict: nil, **)
-          codec = codec_for(engine, level: level, dict: dict)
+        def dialer(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_auto_dict!(auto_dict, dict)
+          codec = codec_for(
+            engine,
+            level: level,
+            dict: dict,
+            auto_dict: normalize_auto_dict(auto_dict),
+          )
           Dialer.new(endpoint, engine, codec)
         end
 
@@ -107,12 +123,41 @@ module OMQ
         # @param dict [String, nil]
         # @return [Codec]
         #
-        def codec_for(engine, level:, dict:)
-          @codecs[engine] ||= Codec.new level: level, dict: dict,
+        def validate_auto_dict!(auto_dict, dict)
+          return unless auto_dict
+
+          if dict
+            raise ArgumentError, "cannot combine auto_dict: and dict:"
+          end
+
+          return if auto_dict == true
+
+          unless auto_dict.is_a?(Hash)
+            raise TypeError, "auto_dict: must be true or a Hash; got #{auto_dict.class}"
+          end
+
+          cap = auto_dict[:capacity]
+          if cap && (cap < 1 || cap > Codec::MAX_DICT_SIZE)
+            raise ArgumentError,
+              "auto_dict capacity #{cap} out of range [1, #{Codec::MAX_DICT_SIZE}]"
+          end
+        end
+
+
+        def normalize_auto_dict(auto_dict)
+          return unless auto_dict
+          return { capacity: Codec::DICT_CAPACITY }.freeze if auto_dict == true
+
+          { capacity: auto_dict[:capacity] || Codec::DICT_CAPACITY }.freeze
+        end
+
+
+        def codec_for(engine, level:, dict:, auto_dict:)
+          @codecs[engine] ||= Codec.new level: level, dict: dict, auto_dict: auto_dict,
             max_message_size: engine.options.max_message_size
 
         rescue Ractor::IsolationError
-          Codec.new level: level, dict: dict,
+          Codec.new level: level, dict: dict, auto_dict: auto_dict,
             max_message_size: engine.options.max_message_size
         end
       end

data/lib/omq/transport/zstd_tcp.rb

@@ -9,7 +9,7 @@
 #   push.connect("zstd+tcp://127.0.0.1:5555", level: -3)
 
 require "omq"
-require "rzstd"
+require "zrip"
 
 require_relative "zstd_tcp/codec"
 require_relative "zstd_tcp/connection"

data/lib/omq/zstd/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module Zstd
-    VERSION = "0.4.2"
+    VERSION = "0.4.4"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-zstd
 version: !ruby/object:Gem::Version
-  version: 0.4.2
+  version: 0.4.4
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,30 +15,31 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
 - !ruby/object:Gem::Dependency
-  name: rzstd
+  name: zrip
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: 0.1.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
-description: Adds zstd+tcp:// endpoint support to OMQ with per-frame Zstd compression,
-  bounded decompression, in-band dictionary shipping, and sender-side dictionary training.
+        version: 0.1.1
+description: Experimental zstd+tcp:// endpoint support for OMQ with per-frame Zstd
+  compression, bounded decompression, in-band dictionary shipping, and sender-side
+  dictionary training.
 email:
 - paddor@gmail.com
 executables: []
@@ -55,7 +56,7 @@ files:
 - lib/omq/transport/zstd_tcp/transport.rb
 - lib/omq/zstd.rb
 - lib/omq/zstd/version.rb
-homepage: https://github.com/paddor/omq-zstd
+homepage: https://github.com/zeromq/omq.rb/tree/main/gems/omq-zstd
 licenses:
 - ISC
 metadata: {}
@@ -66,14 +67,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.3'
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.16
 specification_version: 4
-summary: Zstd+TCP transport for OMQ
+summary: Experimental Zstd+TCP transport for OMQ
 test_files: []