omq-lz4 0.3.1 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b14469227b64cb647b42dbb8495754752ea81654449eab28cae78a0778aa36dd
-  data.tar.gz: 344189a2d0c29ec2d46b310f5607d133bdba87a69fb974be6bb582101f76aacf
+  metadata.gz: 8b849f0045410a07f748583b8647d096110a7dbd632fcb9f3597e216c182cbdd
+  data.tar.gz: 94b0067640c5bef152f7381340b5e620ca1f2c66a42fc7b45de411c35f1ad2d1
 SHA512:
-  metadata.gz: c3ef67d35613434faf125317a61dded584c8d268032eb01830e22a809415ab20b2cf198daa8ee7803d04e88b97d4b39bb6042071a7b18bbf8cff03cb6db06cee
-  data.tar.gz: 49ed574d2b93d2e039160514602cf4f90e125edd9c2b3e05033fbbd5ee1dc1eb098373acaed27c54553e9e6977d6488d3f751b85544ad87a6f0498e5e061cad5
+  metadata.gz: d5c76cf382c5b074d914f0f35cf02b6aef495dea590a6929c8fc41827e67cd07b1a40bec263c2b4ccc71f90510b467d37b638c009a9665a21a0ac6dbc9e154d9
+  data.tar.gz: 8ec8d5da7f5396197af9dfe5ebf903b25ab7463dc51d7f6415c859f02941367080f0adeaf308f4dbb73739c786936fb923e3a04941e3edf3f85bee219df8a2fa

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## [Unreleased]
 
+## 0.3.2 - 2026-07-23
+
+### Changed
+
+- Moved release source to the `zeromq/omq.rb` monorepo.
+- Require `omq ~> 0.28` and Ruby >= 4.0.
+- Require `lz4rip ~> 0.1.1`.
+
 ## 0.3.1 (2026-05-28)
 
 ### Changed
@@ -32,7 +40,7 @@
 
 ### Changed
 
-- use rlz4 ~> 0.5 (ported from lz4_flex to lz4_sys)
+- use the Rust-backed LZ4 block codec with liblz4 bindings
 
 ## 0.1.0 (2026-04-23)
 
@@ -41,8 +49,8 @@
 - Gem skeleton: `omq-lz4.gemspec`, `lib/omq/lz4.rb`,
   `lib/omq/lz4/version.rb`, `lib/omq/lz4/errors.rb`, `Gemfile`,
   `Rakefile`, `.gitignore`, `LICENSE` (ISC), `README.md`, CI and
-  release GitHub Actions workflows, minitest harness. Depends on
-  `rlz4 ~> 0.3` (block API) and `omq ~> 0.23`.
+  release GitHub Actions workflows, minitest harness. Depends on a
+  Rust-backed LZ4 block API and `omq ~> 0.23`.
 - `require "omq/lz4"` succeeds and defines `OMQ::LZ4::VERSION` and
   `OMQ::LZ4::ProtocolError`. No transport behaviour yet — the
   `lz4+tcp://` scheme registration lands in a subsequent milestone.
@@ -138,7 +146,7 @@
     `MIN_COMPRESS_*` thresholds below.
   - `head_to_head.rb` — side-by-side microbenchmark (pure codec)
     and transport throughput of `omq-lz4` vs `omq-zstd`, with and
-    without a shared dictionary. Requires `omq-zstd` + `rzstd` in
+    without a shared dictionary. Requires `omq-zstd` + `zrip` in
     the Gemfile's `:bench` group. Numbers land in README's
     "Head-to-head vs omq-zstd" section.
 - **Compression thresholds tuned** from the sweep:

data/README.md

@@ -2,17 +2,17 @@
 
 [![Gem Version](https://img.shields.io/gem/v/omq-lz4?color=e9573f)](https://rubygems.org/gems/omq-lz4)
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
-[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%204.0-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
 
-LZ4-compressed TCP transport for [OMQ](https://github.com/paddor/omq),
-complementary to [`omq-zstd`](https://github.com/paddor/omq-zstd).
-Pick `lz4+tcp://` instead of `tcp://` or `zstd+tcp://` when you want
-cheap per-message compression with a small per-connection footprint.
+LZ4-compressed TCP transport for [OMQ](https://github.com/zeromq/omq.rb).
+Pick `lz4+tcp://` instead of `tcp://` when you want cheap per-message
+compression with a small per-connection footprint. `omq-lz4` is the
+preferred compressed TCP transport. `omq-zstd` is experimental.
 
 See [RFC.md](RFC.md) for the wire-format specification and
 [CHANGELOG.md](CHANGELOG.md) for release history.
 
-## When to pick `lz4+tcp://` over `zstd+tcp://`
+## Why `lz4+tcp://`
 
 LZ4 has no entropy stage (no Huffman, no FSE), ~16 KiB of encoder state
 per connection, and trades a **worse compression ratio** for
@@ -25,12 +25,14 @@ per connection, and trades a **worse compression ratio** for
 | Memory per connection | ~256 KiB | ~16 KiB + dict |
 | Ratio, 1 KiB JSON no dict | ~45% | ~65% |
 | Ratio, 1 KiB JSON with dict | ~20% | ~35% |
-| Auto-trained dictionaries | yes | no (user-supplied only) |
+| Auto-trained dictionaries | yes | yes |
 
 Pick `omq-lz4` for CPU- or memory-scarce deployments (edge gateways,
 IoT concentrators, high-fanout scenarios where per-connection state
-matters more than ratio). Pick `omq-zstd` for bandwidth-bound
-deployments where CPU is cheap.
+matters more than ratio). It is also the default choice for bandwidth
+bound deployments now that lz4rip supports dictionary training.
+`omq-zstd` remains useful for experiments where the tighter zstd ratio
+is worth extra transport complexity.
 
 ## Install
 
@@ -66,14 +68,19 @@ Both peers must use `lz4+tcp://`. A `tcp://` peer cannot talk to an
 
 Small messages don't compress well on their own. A shared dictionary
 gives 2-5x better ratios on payloads with a common prefix. Supply a
-user-trained dictionary (LZ4 has no auto-training; use `omq-zstd`
-for that):
+user-trained dictionary:
 
 ```ruby
 dict = File.binread("schema.dict")
 push.connect("lz4+tcp://127.0.0.1:5555", dict: dict)
 ```
 
+Or let lz4rip train one from early traffic:
+
+```ruby
+push.connect("lz4+tcp://127.0.0.1:5555", auto_dict: true)
+```
+
 The sender ships the dictionary to the receiver in-band, prefixed
 with the dictionary sentinel (`4C 5A 34 44`, "LZ4D" in ASCII), on
 the first outgoing message. The receiver installs the dictionary
@@ -100,7 +107,7 @@ The receiver bounds decompression by the socket's `max_message_size`
 attempting to send an over-budget message drops the connection.
 `OMQ::SocketDeadError` surfaces on the next `receive`.
 
-Independent of that, the dictionary itself is capped at 8 KiB; a
+Independent of that, the dictionary itself is capped at 8 KiB. A
 larger shipment drops the connection.
 
 ## Wire format
@@ -117,7 +124,7 @@ Every post-handshake ZMTP message part starts with a 4-byte sentinel:
 **Single-block** (`LZ4B`): `sentinel (4) || decompressed_size u64 LE (8) || LZ4 block bytes`.
 12-byte envelope. Raw LZ4 block format (no magic, no descriptor, no
 checksum). `decompressed_size` is required because LZ4 block format
-carries no length prefix; the receiver pre-sizes its output buffer.
+carries no length prefix. The receiver pre-sizes its output buffer.
 
 **Multi-block** (`LZ4M`): same header, followed by a sequence of
 `u32 LE compressed_block_len || LZ4 block bytes` pairs. Each block
@@ -214,10 +221,10 @@ Three regimes visible:
   tighter the ratio, the bigger the win: `zstd 3`'s 3% wire ratio
   translates to a **~17x throughput multiplier** over plain tcp.
 - **1 Gbit**: compressed transports shift from wire-saturated to
-  CPU-limited. `zstd -3` reaches ~75% of wire cap; `zstd 3` only
+  CPU-limited. `zstd -3` reaches ~75% of wire cap. `zstd 3` only
   29% (deep CPU-bound). Both beat plain tcp (which is pinned at
   the wire cap) by **6-8x**. `zstd 3`'s tighter wire no longer
-  helps; there's no wire saturation to trade CPU for.
+  helps. There is no wire saturation to trade CPU for.
 - **Unlimited loopback**: no wire cap. All three are
   CPU-limited. Plain tcp doesn't pay compression CPU, so **skip
   compression on loopback**.
@@ -234,7 +241,7 @@ Or use a `veth` pair in a network namespace so shaping doesn't
 touch your host's real loopback (see `tc-netem(8)`, `ip-netns(8)`).
 
 Full sweeps (8 sizes from 256 B to 512 KiB) for each regime live
-in `bench/head_to_head.rb` output. Run it yourself; the
+in `bench/head_to_head.rb` output. Run it yourself. The
 headline numbers above are stable across repeats but small sizes
 and very large sizes vary a bit run-to-run.
 
@@ -243,12 +250,13 @@ and very large sizes vary a bit run-to-run.
 - Pick **`lz4+tcp://`** for bandwidth-limited links (any real
   network, even 1 Gbit LAN). 6-9x throughput multiplier over
   plain `tcp`, minimal memory (~16 KiB/connection), modest CPU.
-  Ties or beats `zstd -3` at 1 Gbit; loses the ratio race to
+  Ties or beats `zstd -3` at 1 Gbit. Loses the ratio race to
   `zstd 3` at 100 Mbit and below.
 - Pick **`zstd+tcp://` (level >= 3)** when the wire is the
   precious resource (100 Mbit links or slower, WAN, or you're
   paying for egress). **~17x throughput multiplier at 100 Mbit**
-  for 128 KiB messages is hard to argue with.
+  for 128 KiB messages is hard to argue with. Treat this as
+  experimental in Ruby OMQ.
 - Pick **plain `tcp://`** when the link is *not* the bottleneck
   (localhost IPC, loopback, datacenter-fast inter-host
   connections where the bandwidth ceiling is above the CPU's

data/lib/omq/lz4/codec.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require "rlz4"
+require "lz4rip"
 
 require_relative "errors"
 
@@ -62,7 +62,7 @@ module OMQ
       # back to passthrough when compression wouldn't save at least the
       # envelope overhead.
       #
-      # `block_codec` is an RLZ4::BlockCodec, optionally constructed with
+      # `block_codec` is an Lz4rip::BlockCodec, optionally constructed with
       # `dict: bytes`. The codec's dict presence is detected via
       # `#has_dict?` to pick the min-size threshold.
       #
@@ -120,7 +120,7 @@ module OMQ
           block = wire_bytes.byteslice(12, wire_bytes.bytesize - 12)
           begin
             block_codec.decompress(block, decompressed_size: decompressed_size)
-          rescue RLZ4::DecompressError => e
+          rescue Lz4rip::DecompressError => e
             raise ProtocolError, "LZ4B decode failed: #{e.message}"
           end
         when LZ4M_SENTINEL
@@ -213,7 +213,7 @@ module OMQ
             block_decompressed_size = [block_size, remaining].min
             begin
               output << block_codec.decompress(block_data, decompressed_size: block_decompressed_size)
-            rescue RLZ4::DecompressError => e
+            rescue Lz4rip::DecompressError => e
               raise ProtocolError, "LZ4M block decode failed: #{e.message}"
             end
 

data/lib/omq/lz4/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module LZ4
-    VERSION = "0.3.1"
+    VERSION = "0.3.2"
   end
 end

data/lib/omq/transport/lz4_tcp/connection.rb

@@ -12,30 +12,44 @@ module OMQ
       # receive_message route through compression, but everything else
       # (write_command, close, etc.) passes through untouched.
       class Lz4Connection < SimpleDelegator
+        DEFAULT_DICT_CAPACITY = 2048
+        DEFAULT_TRAIN_TRIGGER = 100
+
         # @return [Integer, nil] wire bytesize of the last received
         #   message (sum across parts, compressed sentinel included).
         attr_reader :last_wire_size_in
 
-        def initialize(conn, send_dict_bytes:, max_message_size:)
+        def initialize(conn, send_dict_bytes:, max_message_size:, auto_dict: nil)
           super(conn)
-          @max_message_size  = max_message_size
-
-          # Per-direction state. The send codec is built at construction
-          # with the send-side dictionary (if any) baked in. The receive
-          # codec starts dict-less; if/when a dict shipment arrives on
-          # this direction, it is replaced with a dict-bound codec.
-          @send_dict_bytes   = send_dict_bytes&.b
-          @send_codec        = build_block_codec(@send_dict_bytes)
-          @send_dict_shipped = @send_dict_bytes.nil?  # nothing to ship => "already shipped"
-
-          @recv_codec        = build_block_codec(nil)
-          @recv_dict_bytes   = nil
+          @max_message_size = max_message_size
 
+          @recv_codec       = build_block_codec(nil)
+          @recv_dict_bytes  = nil
           @last_wire_size_in = nil
+
+          if send_dict_bytes
+            @send_dict_bytes   = send_dict_bytes.b
+            @send_codec        = build_block_codec(@send_dict_bytes)
+            @send_dict_shipped = false
+            @trainer           = nil
+          elsif auto_dict
+            @send_dict_bytes   = nil
+            @send_codec        = build_block_codec(nil)
+            @send_dict_shipped = true
+            @trainer           = Lz4rip::DictTrainer.new(auto_dict[:capacity])
+            @train_trigger     = auto_dict[:trigger]
+            @train_msg_count   = 0
+          else
+            @send_dict_bytes   = nil
+            @send_codec        = build_block_codec(nil)
+            @send_dict_shipped = true
+            @trainer           = nil
+          end
         end
 
 
         def send_message(parts)
+          maybe_train!(parts)
           wire = encode_parts(parts)
           ship_send_dict!
           __getobj__.send_message(wire)
@@ -43,6 +57,7 @@ module OMQ
 
 
         def write_message(parts)
+          maybe_train!(parts)
           wire = encode_parts(parts)
           ship_send_dict!
           __getobj__.write_message(wire)
@@ -50,6 +65,7 @@ module OMQ
 
 
         def write_messages(messages)
+          messages.each { |parts| maybe_train!(parts) }
           wires = messages.map { |parts| encode_parts(parts) }
           ship_send_dict!
           __getobj__.write_messages(wires)
@@ -76,10 +92,36 @@ module OMQ
 
         def build_block_codec(dict_bytes)
           if dict_bytes
-            RLZ4::BlockCodec.new(dict: dict_bytes)
+            Lz4rip::BlockCodec.new(dict: dict_bytes)
           else
-            RLZ4::BlockCodec.new
+            Lz4rip::BlockCodec.new
+          end
+        end
+
+
+        def maybe_train!(parts)
+          return unless @trainer
+
+          parts.each do |pt|
+            bytes = pt.is_a?(String) && pt.encoding == Encoding::BINARY ? pt : pt.to_s.b
+            @trainer.add_sample(bytes)
           end
+          @train_msg_count += 1
+          return if @train_msg_count < @train_trigger
+
+          finish_training!
+        end
+
+
+        def finish_training!
+          dict_bytes = @trainer.train
+          @trainer = nil
+
+          return if dict_bytes.empty?
+
+          @send_dict_bytes   = dict_bytes.b.freeze
+          @send_codec        = build_block_codec(@send_dict_bytes)
+          @send_dict_shipped = false
         end
 
 
@@ -137,9 +179,9 @@ module OMQ
 
           dict_bytes = LZ4::Codec.decode_dict_shipment(wire)
           # Replace the no-dict recv codec with a dict-bound one. The
-          # old codec is GC'd. Per rlz4: "fresh codec when dict is
-          # needed" — BlockCodec's dict is a permanent property.
-          @recv_codec      = RLZ4::BlockCodec.new(dict: dict_bytes)
+          # old codec is GC'd. Lz4rip treats dict as a permanent codec
+          # property, so install-time is always a fresh build.
+          @recv_codec      = Lz4rip::BlockCodec.new(dict: dict_bytes)
           @recv_dict_bytes = dict_bytes
         end
 

data/lib/omq/transport/lz4_tcp/transport.rb

@@ -29,9 +29,15 @@ module OMQ
         #   ship on the first outgoing message. If nil, no dict is
         #   shipped (payloads compress without dict or go plaintext
         #   below the min-size threshold).
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training from early traffic (RFC §7.6). Pass `true` for
+        #   defaults (capacity=2048, trigger=100) or a Hash with
+        #   `:capacity` and/or `:trigger` keys. Mutually exclusive
+        #   with `dict:`.
         # @return [Listener]
-        def listener(endpoint, engine, dict: nil, **)
+        def listener(endpoint, engine, dict: nil, auto_dict: nil, **)
           validate_dict!(dict)
+          validate_auto_dict!(auto_dict, dict)
 
           host, port  = parse_endpoint(endpoint)
           lookup_host = normalize_bind_host(host)
@@ -46,7 +52,7 @@ module OMQ
           host_part    = display_host.include?(":") ? "[#{display_host}]" : display_host
           resolved     = "#{SCHEME}://#{host_part}:#{actual_port}"
 
-          Listener.new(resolved, servers, actual_port, engine, dict&.b)
+          Listener.new(resolved, servers, actual_port, engine, dict&.b, normalize_auto_dict(auto_dict))
         end
 
 
@@ -55,10 +61,13 @@ module OMQ
         # @param endpoint [String]
         # @param engine [OMQ::Engine]
         # @param dict [String, nil] user-supplied dictionary bytes.
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. See `listener` for details.
         # @return [Dialer]
-        def dialer(endpoint, engine, dict: nil, **)
+        def dialer(endpoint, engine, dict: nil, auto_dict: nil, **)
           validate_dict!(dict)
-          Dialer.new(endpoint, engine, dict&.b)
+          validate_auto_dict!(auto_dict, dict)
+          Dialer.new(endpoint, engine, dict&.b, normalize_auto_dict(auto_dict))
         end
 
 
@@ -111,6 +120,40 @@ module OMQ
           raise LZ4::ProtocolError,
             "dict size #{size} out of range [1, #{LZ4::Codec::MAX_DICT_SIZE}]"
         end
+
+
+        def validate_auto_dict!(auto_dict, dict)
+          return unless auto_dict
+
+          if dict
+            raise ArgumentError, "cannot combine auto_dict: and dict:"
+          end
+
+          return if auto_dict == true
+
+          unless auto_dict.is_a?(Hash)
+            raise TypeError, "auto_dict: must be true or a Hash; got #{auto_dict.class}"
+          end
+
+          cap = auto_dict[:capacity]
+          if cap && (cap < 1 || cap > LZ4::Codec::MAX_DICT_SIZE)
+            raise ArgumentError,
+              "auto_dict capacity #{cap} out of range [1, #{LZ4::Codec::MAX_DICT_SIZE}]"
+          end
+        end
+
+
+        def normalize_auto_dict(auto_dict)
+          return unless auto_dict
+
+          capacity = Lz4Connection::DEFAULT_DICT_CAPACITY
+          trigger  = Lz4Connection::DEFAULT_TRAIN_TRIGGER
+          if auto_dict.is_a?(Hash)
+            capacity = auto_dict[:capacity] if auto_dict[:capacity]
+            trigger  = auto_dict[:trigger]  if auto_dict[:trigger]
+          end
+          { capacity: capacity, trigger: trigger }.freeze
+        end
       end
 
 
@@ -118,10 +161,11 @@ module OMQ
       class Dialer
         attr_reader :endpoint
 
-        def initialize(endpoint, engine, dict_bytes)
-          @endpoint   = endpoint
-          @engine     = engine
+        def initialize(endpoint, engine, dict_bytes, auto_dict)
+          @endpoint  = endpoint
+          @engine    = engine
           @dict_bytes = dict_bytes
+          @auto_dict = auto_dict
         end
 
 
@@ -144,6 +188,7 @@ module OMQ
             conn,
             send_dict_bytes:  @dict_bytes,
             max_message_size: @engine.options.max_message_size,
+            auto_dict:        @auto_dict,
           )
         end
       end
@@ -153,13 +198,14 @@ module OMQ
       class Listener
         attr_reader :endpoint, :port
 
-        def initialize(endpoint, servers, port, engine, dict_bytes)
-          @endpoint   = endpoint
-          @servers    = servers
-          @port       = port
-          @engine     = engine
+        def initialize(endpoint, servers, port, engine, dict_bytes, auto_dict)
+          @endpoint  = endpoint
+          @servers   = servers
+          @port      = port
+          @engine    = engine
           @dict_bytes = dict_bytes
-          @tasks      = []
+          @auto_dict = auto_dict
+          @tasks     = []
         end
 
 
@@ -168,6 +214,7 @@ module OMQ
             conn,
             send_dict_bytes:  @dict_bytes,
             max_message_size: @engine.options.max_message_size,
+            auto_dict:        @auto_dict,
           )
         end
 

data/lib/omq/transport/lz4_tcp.rb

@@ -11,7 +11,7 @@
 #   push.connect("lz4+tcp://127.0.0.1:5555")
 
 require "omq"
-require "rlz4"
+require "lz4rip"
 
 require_relative "../lz4/errors"
 require_relative "../lz4/codec"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-lz4
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.3.2
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,31 +15,31 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
 - !ruby/object:Gem::Dependency
-  name: rlz4
+  name: lz4rip
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.5'
+        version: 0.1.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.5'
-description: 'Adds lz4+tcp:// endpoint support to OMQ with per-part LZ4 block-format
-  compression, bounded decompression, and in-band dictionary shipping. Complementary
-  to omq-zstd: worse ratio, far faster encode, far smaller per-connection footprint.'
+        version: 0.1.1
+description: Adds lz4+tcp:// endpoint support to OMQ with per-part LZ4 block-format
+  compression, bounded decompression, in-band dictionary shipping, and automatic dictionary
+  training.
 email:
 - paddor@gmail.com
 executables: []
@@ -56,7 +56,7 @@ files:
 - lib/omq/transport/lz4_tcp.rb
 - lib/omq/transport/lz4_tcp/connection.rb
 - lib/omq/transport/lz4_tcp/transport.rb
-homepage: https://github.com/paddor/omq-lz4
+homepage: https://github.com/zeromq/omq.rb/tree/main/gems/omq-lz4
 licenses:
 - ISC
 metadata: {}
@@ -67,14 +67,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.3'
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.10
+rubygems_version: 4.0.16
 specification_version: 4
 summary: LZ4+TCP transport for OMQ
 test_files: []