omq-zstd 0.4.2 → 0.4.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1fcea64306ec7603681d7b3350e7cdf2d22d43487be062e79cc91f51f64aacfb
-  data.tar.gz: ac5d8d88deb3f90ccf97be757d384612274f4d742b042b79bc949a47ea7d0d3c
+  metadata.gz: 5a3ed977d7fa2b57910a209fd3db1ca683634a9ccc2d299cd3dd0f95c0030d93
+  data.tar.gz: 4d439082a1142c335538c3ea3698cc629e6dc9fa406335386171d6457ad2a9e6
 SHA512:
-  metadata.gz: 8e5a04f9ffdb46d130af145fa7025c544cc1709e7711cd92180b0408923667e61d70098efc4679c7b0d2617a3acde40b91c9ecaa1b5abb9e56cadc4a34cd8294
-  data.tar.gz: 25ab87f733a74257a9b2d772d96a76be134b60804ce2796fa0a95710334eb0552f973fab4a7b04442f6d9cce8091da008c5ed17e6868e925b942fdc4dfefd0f6
+  metadata.gz: a68d93c3c116a9fe508844058515f74733a674c6a8a085149b8bff2a8cfb089793509a5d3aed54cc56c3730ecd8386e6eb12ac500cad734e58d8976b6cc674bd
+  data.tar.gz: 9eb6751a7682a0ebefd9680b9c48f8a1f6c31041e2f276e87064054d6d7094145a8e6a4776579ea33b9d4c10eacd6d4ce6817c6c4cc3e0d4ec1401ecd659a01c

data/DESIGN.md

@@ -67,7 +67,7 @@ One Codec per Dialer or Listener. Shared across all connections of
 that endpoint. Owns:
 
 - **Compression**: `#compress_parts(parts)` with identity cache
-- **Training**: sample collection, `RZstd::Dictionary.train`, dict ID patching
+- **Training**: sample collection, `Zrip::DictTrainer`, dict ID patching
 - **Send dict**: `#send_dict_bytes` — the trained or user-supplied dict bytes
 
 ### ZstdConnection (per-connection)
@@ -124,7 +124,7 @@ end
 - **Sample size cap**: frames > 1024 bytes are skipped (dictionaries primarily benefit small frames)
 - **Dict capacity**: 8 KiB (conservative; Zstd recommends ~100:1 sample-to-dict ratio)
 - **Dict ID patching**: auto-trained dicts get a random ID in the user range (32768..2^31-1) to avoid collisions with Zstd's built-in dict IDs
-- **Training failure**: if `RZstd::Dictionary.train` raises, training is disabled permanently for the socket. No retry.
+- **Training failure**: if `Zrip::DictTrainer#train` raises, training is disabled permanently for the socket. No retry.
 
 ## Frame dispatch
 

data/README.md

@@ -2,13 +2,20 @@
 
 [![Gem Version](https://img.shields.io/gem/v/omq-zstd?color=e9573f)](https://rubygems.org/gems/omq-zstd)
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
-[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%204.0-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+
+Experimental Zstandard-compressed TCP transport for
+[OMQ](https://github.com/zeromq/omq.rb).
+
+Use [`omq-lz4`](../omq-lz4) for new compressed TCP work. OMQ.rs removed
+`zstd+tcp://` to reduce transport complexity after lz4rip gained
+dictionary training. This gem remains for research, comparison, and
+cases where zstd ratio matters more than transport simplicity.
 
-Zstandard-compressed TCP transport for [OMQ](https://github.com/paddor/omq).
 Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
 compressed per-part with [Zstandard](https://github.com/facebook/zstd).
 Compression is intrinsic to the transport: no negotiation, no socket option,
-no payload changes. The ZMTP handshake runs over plain TCP; only
+no payload changes. The ZMTP handshake runs over plain TCP. Only
 post-handshake message parts are compressed.
 
 See [RFC.md](RFC.md) for the wire-format specification and
@@ -170,7 +177,7 @@ It is **not** worth it for:
 `require "omq/zstd"` registers the `zstd+tcp` scheme on
 `OMQ::Engine.transports`. A `zstd+tcp` socket builds a per-engine
 `Codec` (one Zstd dictionary instance shared across all the socket's
-connections; fan-out compresses each part exactly once). Each accepted
+connections. Fan-out compresses each part exactly once). Each accepted
 or dialed TCP connection is wrapped in `ZstdConnection`, a
 `SimpleDelegator` over the underlying ZMTP connection that intercepts
 `#send_message` / `#write_message` / `#receive_message`. Message parts

data/RFC.md

@@ -338,23 +338,32 @@ in shared training; they use their own dictionary independently.
 
 ### 7.5 Automatic dictionary training
 
-A sender MAY train a dictionary automatically from early traffic:
-
-1. Buffer plaintext samples from the first messages. Samples larger
-   than **2048 bytes** SHOULD be skipped; dictionaries primarily
-   benefit small frames.
-2. When the buffer reaches **1000 samples** OR **100 KiB** of
-   plaintext (whichever comes first), train a Zstandard dictionary from
-   the buffered samples and discard the buffer.
-3. The recommended dictionary capacity (training target size) is
-   **2 KiB**.
-4. Ship the trained dictionary via a dictionary message (Sec. 7.1) on
-   every connection, before any compressed part that uses it.
-5. Switch to dictionary-bound compression for all subsequent parts.
-
-If training fails (the sample set was too small or too uniform), the
-sender MUST stay in no-dictionary mode for the rest of the socket's
-lifetime. It MUST NOT retry training.
+A sender MAY train a dictionary automatically from early traffic.
+The training algorithm, trigger condition, and parameters are
+implementation choices. The following wire-level constraints MUST be
+respected:
+
+- A trained dictionary MUST NOT exceed the maximum dictionary size
+  (Sec. 7.2: 8 KiB).
+- A sender MUST ship at most one dictionary per direction per
+  connection (Sec. 7.2).
+- If training fails (the sample set was too small or too uniform),
+  the sender MUST stay in no-dictionary mode for the rest of the
+  socket's lifetime. It MUST NOT retry training.
+
+The following parameters are RECOMMENDED for implementations that
+support auto-training:
+
+| Parameter              | Recommended value                        |
+|------------------------|------------------------------------------|
+| Dictionary capacity    | 2 KiB                                    |
+| Training trigger       | 1000 samples OR 100 KiB (first reached)  |
+| Max sample length      | 2048 bytes                               |
+| Training algorithm     | FastCOVER                                |
+
+Whether auto-training is enabled by default is an implementation
+choice. Applications MAY disable it or supply an out-of-band
+dictionary instead.
 
 ### 7.6 Dictionary ID
 

data/lib/omq/transport/zstd_tcp/codec.rb

@@ -28,10 +28,10 @@ module OMQ
 
           # Start with a no-dict FrameCodec. Once a dict is configured
           # (either via the `dict:` kwarg or via auto-training), this is
-          # replaced with a fresh dict-bound FrameCodec — per rzstd 0.4,
+          # replaced with a fresh dict-bound FrameCodec. In Zrip, the
           # dict is a permanent property of the codec, so swapping dicts
           # means constructing a new one.
-          @send_codec      = RZstd::FrameCodec.new(level: @level)
+          @send_codec      = Zrip::FrameCodec.new(level: @level)
           @send_dict_bytes = nil
 
           @training      = dict.nil?
@@ -102,18 +102,16 @@ module OMQ
           return unless @train_samples.size >= TRAIN_MAX_SAMPLES ||
                         @train_bytes >= TRAIN_MAX_BYTES
 
-          begin
-            trained = RZstd::Dictionary.train(@train_samples, capacity: DICT_CAPACITY)
-          rescue RuntimeError
-            @training = false
-            @train_samples = nil
-            return
-          end
+          trainer = Zrip::DictTrainer.new(DICT_CAPACITY)
+          @train_samples.each { |s| trainer.add_sample(s) }
+          trained_bytes = trainer.train
 
           @training = false
           @train_samples = nil
 
-          patched = patch_auto_dict_id(trained.bytes)
+          return if trained_bytes.empty?
+
+          patched = patch_auto_dict_id(trained_bytes)
           install_send_dict(patched)
         end
 
@@ -136,9 +134,9 @@ module OMQ
           end
 
           # Replace the no-dict send codec with a fresh dict-bound one;
-          # the old codec is GC'd. rzstd 0.4 treats dict as a permanent
-          # codec property, so install-time is always a fresh build.
-          @send_codec      = RZstd::FrameCodec.new(dict: bytes, level: @level)
+          # the old codec is GC'd. Zrip treats dict as a permanent codec
+          # property, so install-time is always a fresh build.
+          @send_codec      = Zrip::FrameCodec.new(dict: bytes, level: @level)
           @send_dict_bytes = bytes
         end
 

data/lib/omq/transport/zstd_tcp/connection.rb

@@ -15,12 +15,11 @@ module OMQ
           super(conn)
           @codec              = codec
           @dict_shipped       = false
-          # rzstd 0.4: FrameCodec is the decoder. Starts no-dict; when a
-          # dict shipment arrives on this direction, we build a fresh
-          # dict-bound FrameCodec and replace this one. Level is a
-          # compression parameter only — not consulted on decompress —
-          # so we don't carry the send-side level through to recv.
-          @recv_codec         = RZstd::FrameCodec.new
+          # FrameCodec is the decoder. Starts no-dict; when a dict
+          # shipment arrives on this direction, we build a fresh
+          # dict-bound FrameCodec and replace this one.
+          @recv_codec         = Zrip::FrameCodec.new
+          @recv_no_dict_codec = @recv_codec
           @recv_dict_bytes    = nil
           @last_wire_size_in  = nil
         end
@@ -128,16 +127,24 @@ module OMQ
 
           decompress_opts = budget ? { max_output_size: budget } : {}
 
-          @recv_codec.decompress(wire, **decompress_opts)
-        rescue RZstd::DecompressError => e
+          codec = frame_has_dict_id?(wire) ? @recv_codec : @recv_no_dict_codec
+          codec.decompress(wire, **decompress_opts)
+        rescue Zrip::DecompressError => e
           raise ProtocolError, "decompression failed: #{e.message}"
-        rescue RZstd::MissingContentSizeError => e
+        rescue Zrip::MissingContentSizeError => e
           raise ProtocolError, "Zstd frame missing Frame_Content_Size (#{e.message})"
-        rescue RZstd::OutputSizeLimitError => e
+        rescue Zrip::OutputSizeLimitError => e
           raise ProtocolError, "declared FCS exceeds limit (#{e.message})"
         end
 
 
+        def frame_has_dict_id?(wire)
+          return false if wire.bytesize < 5
+
+          (wire.getbyte(4) & 0x03) != 0
+        end
+
+
         def install_recv_dict(wire)
           if wire.bytesize < 8
             raise ProtocolError, "dict frame too short"
@@ -147,10 +154,7 @@ module OMQ
             raise ProtocolError, "dict exceeds #{Codec::MAX_DICT_SIZE} bytes"
           end
 
-          # Replace the no-dict recv codec with a fresh dict-bound one;
-          # the old codec is GC'd (rzstd 0.4: dict is a permanent codec
-          # property).
-          @recv_codec      = RZstd::FrameCodec.new(dict: wire.b)
+          @recv_codec      = Zrip::FrameCodec.new(dict: wire.b)
           @recv_dict_bytes = wire.b
         end
 

data/lib/omq/transport/zstd_tcp.rb

@@ -9,7 +9,7 @@
 #   push.connect("zstd+tcp://127.0.0.1:5555", level: -3)
 
 require "omq"
-require "rzstd"
+require "zrip"
 
 require_relative "zstd_tcp/codec"
 require_relative "zstd_tcp/connection"

data/lib/omq/zstd/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module Zstd
-    VERSION = "0.4.2"
+    VERSION = "0.4.3"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-zstd
 version: !ruby/object:Gem::Version
-  version: 0.4.2
+  version: 0.4.3
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,30 +15,31 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
 - !ruby/object:Gem::Dependency
-  name: rzstd
+  name: zrip
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: 0.1.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
-description: Adds zstd+tcp:// endpoint support to OMQ with per-frame Zstd compression,
-  bounded decompression, in-band dictionary shipping, and sender-side dictionary training.
+        version: 0.1.1
+description: Experimental zstd+tcp:// endpoint support for OMQ with per-frame Zstd
+  compression, bounded decompression, in-band dictionary shipping, and sender-side
+  dictionary training.
 email:
 - paddor@gmail.com
 executables: []
@@ -55,7 +56,7 @@ files:
 - lib/omq/transport/zstd_tcp/transport.rb
 - lib/omq/zstd.rb
 - lib/omq/zstd/version.rb
-homepage: https://github.com/paddor/omq-zstd
+homepage: https://github.com/zeromq/omq.rb/tree/main/gems/omq-zstd
 licenses:
 - ISC
 metadata: {}
@@ -66,14 +67,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.3'
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.16
 specification_version: 4
-summary: Zstd+TCP transport for OMQ
+summary: Experimental Zstd+TCP transport for OMQ
 test_files: []