omq-zstd 0.4.3 → 0.4.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5a3ed977d7fa2b57910a209fd3db1ca683634a9ccc2d299cd3dd0f95c0030d93
-  data.tar.gz: 4d439082a1142c335538c3ea3698cc629e6dc9fa406335386171d6457ad2a9e6
+  metadata.gz: 4bd678bf2436ab0d01597aaff7d536957432abcd571c36b21b6d9d82eff12061
+  data.tar.gz: df584bef85cac3e3e031990bb511675bdb9b4aca50fc9975f869666d26154e47
 SHA512:
-  metadata.gz: a68d93c3c116a9fe508844058515f74733a674c6a8a085149b8bff2a8cfb089793509a5d3aed54cc56c3730ecd8386e6eb12ac500cad734e58d8976b6cc674bd
-  data.tar.gz: 9eb6751a7682a0ebefd9680b9c48f8a1f6c31041e2f276e87064054d6d7094145a8e6a4776579ea33b9d4c10eacd6d4ce6817c6c4cc3e0d4ec1401ecd659a01c
+  metadata.gz: 83b6c89aec844c6c14d5afd2161173c3ef9aa915a821d5a05c78610637994fdd68a8d240ef05494136f02c7e4ec27363c5da48f8392dd017d5b728da53114a58
+  data.tar.gz: 87c79b3d3a07f4c6e42f823af146be14ab2c91770c3df221a7372e6c4554130fe17d5a4eebf66e9a9ec27b45ee581916ff3eb32362d10d1e21ba5c390d5373d2

data/README.md

@@ -7,10 +7,9 @@
 Experimental Zstandard-compressed TCP transport for
 [OMQ](https://github.com/zeromq/omq.rb).
 
-Use [`omq-lz4`](../omq-lz4) for new compressed TCP work. OMQ.rs removed
-`zstd+tcp://` to reduce transport complexity after lz4rip gained
-dictionary training. This gem remains for research, comparison, and
-cases where zstd ratio matters more than transport simplicity.
+Use [`omq-lz4`](../omq-lz4) for CPU- or memory-sensitive compressed TCP
+work. This gem remains for research, comparison, OMQ.rs parity, and cases
+where zstd ratio matters more than transport simplicity.
 
 Pick `zstd+tcp://` instead of `tcp://` and every message part on the wire is
 compressed per-part with [Zstandard](https://github.com/facebook/zstd).
@@ -80,11 +79,17 @@ The sender ships the dictionary to the receiver in-band as a one-shot
 single-part message prefixed with the dictionary sentinel
 (`37 A4 30 EC`), so the receiver does not need a copy on disk.
 
-**Auto-trained dictionary** (zero config, the default when no `dict:` is
-passed): the sender collects up to 1000 samples or 100 KiB (whichever hits
-first), skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary,
-ships it inline, and switches to dictionary mode. Until then, payloads are
-compressed without a dictionary or sent plaintext when below the threshold.
+**Auto-trained dictionary** (opt-in):
+
+```ruby
+push.connect("zstd+tcp://127.0.0.1:5555", auto_dict: true)
+```
+
+The sender collects up to 1000 samples or 100 KiB (whichever hits first),
+skipping samples larger than 2048 bytes. It trains a 2 KiB dictionary, ships
+it inline, and switches to dictionary mode. Until then, payloads are compressed
+without a dictionary or sent plaintext when below the threshold. Automatic
+dictionary training is off by default.
 
 ### Compression thresholds
 
@@ -152,6 +157,7 @@ transport layer. They are not delivered to the application.
 | Train max bytes | 100 KiB |
 | Train max sample length | 2048 B |
 | Dictionary capacity | 2 KiB |
+| Auto dictionary training | Off by default |
 
 ## When to use it
 

data/RFC.md

@@ -265,21 +265,26 @@ exceeds the limit MUST be rejected before decoder invocation.
 ### 7.1 Dictionary message format
 
 A dictionary is shipped as a **single-part ZMTP message** (no MORE flag)
-whose body begins with the dictionary sentinel:
+whose body is exactly a Zstandard dictionary blob in ZDICT format. ZDICT
+blobs begin with the dictionary magic `37 A4 30 EC`; this transport uses
+those first 4 bytes as the dictionary sentinel:
 
 ```
 +------------------+------------------------+
-| 37 A4 30 EC      | dictionary bytes       |
-| (4 bytes)        | (D bytes)              |
+| 37 A4 30 EC      | rest of ZDICT blob     |
+| (4 bytes)        | (D - 4 bytes)          |
 +------------------+------------------------+
 ```
 
-The sentinel `37 A4 30 EC` is specific to this specification and has no
-relationship to Zstandard's internals. It was chosen to avoid collision
-with the Zstandard frame magic and the uncompressed sentinel.
+The sentinel `37 A4 30 EC` is the standard Zstandard dictionary magic
+(`DICT_MAGIC`, little-endian). It is intentionally reused as this
+transport's dictionary discriminator because a ZDICT blob already carries
+it and it does not collide with the Zstandard frame magic or the
+uncompressed sentinel.
 
-The remaining `D` bytes are the raw dictionary as it should be passed
-to the Zstandard decoder's dictionary-load operation.
+The complete `D` bytes, including the leading `37 A4 30 EC` magic, are
+the raw dictionary as it should be passed to the Zstandard decoder's
+dictionary-load operation.
 
 ### 7.2 Constraints
 
@@ -287,9 +292,10 @@ to the Zstandard decoder's dictionary-load operation.
   not set on the frame header). A dictionary sentinel in a multipart
   message's non-final or non-only part is a protocol error.
 
-- A dictionary message MUST NOT exceed **8 KiB** total (sentinel +
-  dictionary bytes). A receiver that receives a dictionary message
-  larger than 8 KiB MUST close the connection.
+- A dictionary message MUST NOT exceed **8 KiB** total. Since the
+  message body is exactly the ZDICT blob, this is also the maximum
+  dictionary size. A receiver that receives a dictionary message larger
+  than 8 KiB MUST close the connection.
 
 - A sender MUST send at most **one** dictionary message per direction
   per connection. A receiver that receives a second dictionary message
@@ -305,9 +311,9 @@ to the Zstandard decoder's dictionary-load operation.
 When the receiver encounters a dictionary part:
 
 1. Validate the constraints in Sec. 7.2.
-2. Strip the 4-byte sentinel.
-3. Install the remaining bytes as the decompression dictionary for this
-   connection.
+2. Install the complete message body as the decompression dictionary for
+   this connection. The leading `37 A4 30 EC` bytes are part of the ZDICT
+   blob and MUST NOT be stripped.
 4. Discard the message. It is not delivered to the application.
 
 If all parts of a ZMTP message are dictionary parts (which is always
@@ -361,9 +367,8 @@ support auto-training:
 | Max sample length      | 2048 bytes                               |
 | Training algorithm     | FastCOVER                                |
 
-Whether auto-training is enabled by default is an implementation
-choice. Applications MAY disable it or supply an out-of-band
-dictionary instead.
+Auto-training SHOULD be disabled by default. Applications MAY enable it
+explicitly or supply an out-of-band dictionary instead.
 
 ### 7.6 Dictionary ID
 

data/lib/omq/transport/zstd_tcp/codec.rb

@@ -22,9 +22,14 @@ module OMQ
         attr_reader :send_dict_bytes, :max_message_size
 
 
-        def initialize(level:, dict: nil, max_message_size: nil)
+        def initialize(level:, dict: nil, max_message_size: nil, auto_dict: nil)
           @level            = level
           @max_message_size = max_message_size
+          @dict_capacity    = if auto_dict.is_a?(Hash) && auto_dict[:capacity]
+                                auto_dict[:capacity]
+                              else
+                                DICT_CAPACITY
+                              end
 
           # Start with a no-dict FrameCodec. Once a dict is configured
           # (either via the `dict:` kwarg or via auto-training), this is
@@ -34,7 +39,7 @@ module OMQ
           @send_codec      = Zrip::FrameCodec.new(level: @level)
           @send_dict_bytes = nil
 
-          @training      = dict.nil?
+          @training      = !!auto_dict && dict.nil?
           @train_samples = []
           @train_bytes   = 0
 
@@ -102,7 +107,7 @@ module OMQ
           return unless @train_samples.size >= TRAIN_MAX_SAMPLES ||
                         @train_bytes >= TRAIN_MAX_BYTES
 
-          trainer = Zrip::DictTrainer.new(DICT_CAPACITY)
+          trainer = Zrip::DictTrainer.new(@dict_capacity)
           @train_samples.each { |s| trainer.add_sample(s) }
           trained_bytes = trainer.train
 

data/lib/omq/transport/zstd_tcp/transport.rb

@@ -24,10 +24,18 @@ module OMQ
         # @param engine [Engine]
         # @param level [Integer] Zstd compression level
         # @param dict [String, nil] user-supplied dictionary bytes
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. Pass `true` for defaults or `{ capacity: N }`.
         # @return [Listener]
         #
-        def listener(endpoint, engine, level: -3, dict: nil, **)
-          codec = codec_for(engine, level: level, dict: dict)
+        def listener(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_auto_dict!(auto_dict, dict)
+          codec = codec_for(
+            engine,
+            level: level,
+            dict: dict,
+            auto_dict: normalize_auto_dict(auto_dict),
+          )
 
           host, port = parse_endpoint(endpoint)
           host       = normalize_bind_host(host)
@@ -52,10 +60,18 @@ module OMQ
         # @param engine [Engine]
         # @param level [Integer] Zstd compression level
         # @param dict [String, nil] user-supplied dictionary bytes
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. Pass `true` for defaults or `{ capacity: N }`.
         # @return [Dialer]
         #
-        def dialer(endpoint, engine, level: -3, dict: nil, **)
-          codec = codec_for(engine, level: level, dict: dict)
+        def dialer(endpoint, engine, level: -3, dict: nil, auto_dict: nil, **)
+          validate_auto_dict!(auto_dict, dict)
+          codec = codec_for(
+            engine,
+            level: level,
+            dict: dict,
+            auto_dict: normalize_auto_dict(auto_dict),
+          )
           Dialer.new(endpoint, engine, codec)
         end
 
@@ -107,12 +123,41 @@ module OMQ
         # @param dict [String, nil]
         # @return [Codec]
         #
-        def codec_for(engine, level:, dict:)
-          @codecs[engine] ||= Codec.new level: level, dict: dict,
+        def validate_auto_dict!(auto_dict, dict)
+          return unless auto_dict
+
+          if dict
+            raise ArgumentError, "cannot combine auto_dict: and dict:"
+          end
+
+          return if auto_dict == true
+
+          unless auto_dict.is_a?(Hash)
+            raise TypeError, "auto_dict: must be true or a Hash; got #{auto_dict.class}"
+          end
+
+          cap = auto_dict[:capacity]
+          if cap && (cap < 1 || cap > Codec::MAX_DICT_SIZE)
+            raise ArgumentError,
+              "auto_dict capacity #{cap} out of range [1, #{Codec::MAX_DICT_SIZE}]"
+          end
+        end
+
+
+        def normalize_auto_dict(auto_dict)
+          return unless auto_dict
+          return { capacity: Codec::DICT_CAPACITY }.freeze if auto_dict == true
+
+          { capacity: auto_dict[:capacity] || Codec::DICT_CAPACITY }.freeze
+        end
+
+
+        def codec_for(engine, level:, dict:, auto_dict:)
+          @codecs[engine] ||= Codec.new level: level, dict: dict, auto_dict: auto_dict,
             max_message_size: engine.options.max_message_size
 
         rescue Ractor::IsolationError
-          Codec.new level: level, dict: dict,
+          Codec.new level: level, dict: dict, auto_dict: auto_dict,
             max_message_size: engine.options.max_message_size
         end
       end

data/lib/omq/zstd/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module Zstd
-    VERSION = "0.4.3"
+    VERSION = "0.4.5"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-zstd
 version: !ruby/object:Gem::Version
-  version: 0.4.3
+  version: 0.4.5
 platform: ruby
 authors:
 - Patrik Wenger
@@ -29,14 +29,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.1.1
+        version: 0.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: 0.1.1
+        version: 0.2.0
 description: Experimental zstd+tcp:// endpoint support for OMQ with per-frame Zstd
   compression, bounded decompression, in-band dictionary shipping, and sender-side
   dictionary training.