omq-lz4 0.3.0 → 0.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4217a939f757a07b0e3e2ad1a905d11ea3cd017e54baa2f059a75f304386bdab
-  data.tar.gz: 1d0f4e4076e0cf02291b3f714fe20c721b8e05e0d4d17af4fbc8662e83897319
+  metadata.gz: 8b849f0045410a07f748583b8647d096110a7dbd632fcb9f3597e216c182cbdd
+  data.tar.gz: 94b0067640c5bef152f7381340b5e620ca1f2c66a42fc7b45de411c35f1ad2d1
 SHA512:
-  metadata.gz: a963f09065ef7a019a8b766a00093e5ef84ba9489fed66d3524da5e8a0d8d956d27cd01534926ac1cb3f5c5088439da9e0db5c5f71c0b7b58b7b90db8ccfc8bf
-  data.tar.gz: 86e7b5fe26c080a13a6b5c5c44b0e0a250bc45d650b2a9f76f4a80330209d6e24374b8a6412dd28118cb971619f56d3bede7a5ed937c91c5a80fd520ab17b5ac
+  metadata.gz: d5c76cf382c5b074d914f0f35cf02b6aef495dea590a6929c8fc41827e67cd07b1a40bec263c2b4ccc71f90510b467d37b638c009a9665a21a0ac6dbc9e154d9
+  data.tar.gz: 8ec8d5da7f5396197af9dfe5ebf903b25ab7463dc51d7f6415c859f02941367080f0adeaf308f4dbb73739c786936fb923e3a04941e3edf3f85bee219df8a2fa

data/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## [Unreleased]
+
+## 0.3.2 - 2026-07-23
+
+### Changed
+
+- Moved release source to the `zeromq/omq.rb` monorepo.
+- Require `omq ~> 0.28` and Ruby >= 4.0.
+- Require `lz4rip ~> 0.1.1`.
+
+## 0.3.1 (2026-05-28)
+
+### Changed
+
+- `MIN_COMPRESS_WITH_DICT`: raised from 32 to 128. The previous value
+  was too aggressive; 128 leaves a safer margin above the measured
+  crossover.
+
 ## 0.3.0 (2026-05-11)
 
 ### Added
@@ -22,7 +40,7 @@
 
 ### Changed
 
-- use rlz4 ~> 0.5 (ported from lz4_flex to lz4_sys)
+- use the Rust-backed LZ4 block codec with liblz4 bindings
 
 ## 0.1.0 (2026-04-23)
 
@@ -31,8 +49,8 @@
 - Gem skeleton: `omq-lz4.gemspec`, `lib/omq/lz4.rb`,
   `lib/omq/lz4/version.rb`, `lib/omq/lz4/errors.rb`, `Gemfile`,
   `Rakefile`, `.gitignore`, `LICENSE` (ISC), `README.md`, CI and
-  release GitHub Actions workflows, minitest harness. Depends on
-  `rlz4 ~> 0.3` (block API) and `omq ~> 0.23`.
+  release GitHub Actions workflows, minitest harness. Depends on a
+  Rust-backed LZ4 block API and `omq ~> 0.23`.
 - `require "omq/lz4"` succeeds and defines `OMQ::LZ4::VERSION` and
   `OMQ::LZ4::ProtocolError`. No transport behaviour yet — the
   `lz4+tcp://` scheme registration lands in a subsequent milestone.
@@ -128,7 +146,7 @@
     `MIN_COMPRESS_*` thresholds below.
   - `head_to_head.rb` — side-by-side microbenchmark (pure codec)
     and transport throughput of `omq-lz4` vs `omq-zstd`, with and
-    without a shared dictionary. Requires `omq-zstd` + `rzstd` in
+    without a shared dictionary. Requires `omq-zstd` + `zrip` in
     the Gemfile's `:bench` group. Numbers land in README's
     "Head-to-head vs omq-zstd" section.
 - **Compression thresholds tuned** from the sweep:

data/README.md

@@ -2,21 +2,21 @@
 
 [![Gem Version](https://img.shields.io/gem/v/omq-lz4?color=e9573f)](https://rubygems.org/gems/omq-lz4)
 [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](LICENSE)
-[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%203.3-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
+[![Ruby](https://img.shields.io/badge/Ruby-%3E%3D%204.0-CC342D?logo=ruby&logoColor=white)](https://www.ruby-lang.org)
 
-> [RFC.md](RFC.md) for the wire-format spec and
-> [CHANGELOG.md](CHANGELOG.md) for what's in.
+LZ4-compressed TCP transport for [OMQ](https://github.com/zeromq/omq.rb).
+Pick `lz4+tcp://` instead of `tcp://` when you want cheap per-message
+compression with a small per-connection footprint. `omq-lz4` is the
+preferred compressed TCP transport. `omq-zstd` is experimental.
 
-LZ4-compressed TCP transport for [OMQ](https://github.com/paddor/omq),
-complementary to [`omq-zstd`](https://github.com/paddor/omq-zstd).
-Pick `lz4+tcp://` instead of `tcp://` or `zstd+tcp://` when you want
-cheap per-message compression with a small per-connection footprint.
+See [RFC.md](RFC.md) for the wire-format specification and
+[CHANGELOG.md](CHANGELOG.md) for release history.
 
-## When to pick `lz4+tcp://` over `zstd+tcp://`
+## Why `lz4+tcp://`
 
 LZ4 has no entropy stage (no Huffman, no FSE), ~16 KiB of encoder state
 per connection, and trades a **worse compression ratio** for
-**~4–8× faster encode** and **~3× less memory per connection**.
+**~4-8x faster encode** and **~3x less memory per connection**.
 
 | | `zstd+tcp://` | `lz4+tcp://` |
 |---|---|---|
@@ -25,12 +25,14 @@ per connection, and trades a **worse compression ratio** for
 | Memory per connection | ~256 KiB | ~16 KiB + dict |
 | Ratio, 1 KiB JSON no dict | ~45% | ~65% |
 | Ratio, 1 KiB JSON with dict | ~20% | ~35% |
-| Auto-trained dictionaries | ✓ | — (user-supplied only) |
+| Auto-trained dictionaries | yes | yes |
 
 Pick `omq-lz4` for CPU- or memory-scarce deployments (edge gateways,
 IoT concentrators, high-fanout scenarios where per-connection state
-matters more than ratio). Pick `omq-zstd` for bandwidth-bound
-deployments where CPU is cheap.
+matters more than ratio). It is also the default choice for bandwidth
+bound deployments now that lz4rip supports dictionary training.
+`omq-zstd` remains useful for experiments where the tighter zstd ratio
+is worth extra transport complexity.
 
 ## Install
 
@@ -60,27 +62,30 @@ pull.receive  # => ["hello, compressed world"]
 ```
 
 Both peers must use `lz4+tcp://`. A `tcp://` peer cannot talk to an
-`lz4+tcp://` peer — they speak different transports.
+`lz4+tcp://` peer. They speak different transports.
 
 ### Dictionary compression
 
 Small messages don't compress well on their own. A shared dictionary
-gives 2–5× better ratios on payloads with a common prefix. Supply a
-user-trained dictionary (LZ4 has no auto-training — use `omq-zstd`
-for that):
+gives 2-5x better ratios on payloads with a common prefix. Supply a
+user-trained dictionary:
 
 ```ruby
 dict = File.binread("schema.dict")
 push.connect("lz4+tcp://127.0.0.1:5555", dict: dict)
 ```
 
+Or let lz4rip train one from early traffic:
+
+```ruby
+push.connect("lz4+tcp://127.0.0.1:5555", auto_dict: true)
+```
+
 The sender ships the dictionary to the receiver in-band, prefixed
 with the dictionary sentinel (`4C 5A 34 44`, "LZ4D" in ASCII), on
 the first outgoing message. The receiver installs the dictionary
 and decompresses subsequent messages against it. Dictionary size
-is capped at **8 KiB** — tighter than `omq-zstd`'s 64 KiB cap, to
-let constrained peers accept shipments without allocating tens of
-KB of scratch.
+is capped at **8 KiB** (same cap as `omq-zstd`).
 
 ### Compression thresholds
 
@@ -89,7 +94,7 @@ To avoid pessimizing tiny frames, the sender skips compression below:
 | Mode            | Threshold |
 |-----------------|-----------|
 | No dictionary   | 512 B     |
-| With dictionary | 32 B      |
+| With dictionary | 128 B     |
 
 Below the threshold the part is sent uncompressed (4-byte zero
 sentinel + plaintext).
@@ -99,14 +104,54 @@ sentinel + plaintext).
 The receiver bounds decompression by the socket's `max_message_size`
 (the same knob you'd use on a plain `tcp://` socket). It caps the
 **total decompressed size of all parts in a single message**. A peer
-attempting to send an over-budget message drops the connection —
+attempting to send an over-budget message drops the connection.
 `OMQ::SocketDeadError` surfaces on the next `receive`.
 
-Independent of that, the dictionary itself is capped at 8 KiB; a
+Independent of that, the dictionary itself is capped at 8 KiB. A
 larger shipment drops the connection.
 
-See the plan roadmap ([../OMQ-LZ4.plan](../OMQ-LZ4.plan)) for
-history and open questions.
+## Wire format
+
+Every post-handshake ZMTP message part starts with a 4-byte sentinel:
+
+| Sentinel (hex) | ASCII | Meaning |
+|---|---|---|
+| `00 00 00 00` | (none) | Uncompressed plaintext |
+| `4C 5A 34 42` | `LZ4B` | LZ4-compressed single block |
+| `4C 5A 34 4D` | `LZ4M` | LZ4-compressed multi-block |
+| `4C 5A 34 44` | `LZ4D` | Dictionary shipment |
+
+**Single-block** (`LZ4B`): `sentinel (4) || decompressed_size u64 LE (8) || LZ4 block bytes`.
+12-byte envelope. Raw LZ4 block format (no magic, no descriptor, no
+checksum). `decompressed_size` is required because LZ4 block format
+carries no length prefix. The receiver pre-sizes its output buffer.
+
+**Multi-block** (`LZ4M`): same header, followed by a sequence of
+`u32 LE compressed_block_len || LZ4 block bytes` pairs. Each block
+decompresses independently at up to 1 GiB. Used for parts exceeding
+the single-block size cap.
+
+**Dictionary shipment** (`LZ4D`): `sentinel (4) || dict bytes (1..8192)`.
+Single-part ZMTP message consumed by the transport, not delivered
+to the application. At most one per direction per connection.
+
+Any other leading 4 bytes close the connection.
+
+## Constants
+
+| Constant | Value |
+|---|---|
+| Scheme | `lz4+tcp` |
+| Uncompressed sentinel | `00 00 00 00` |
+| Single-block sentinel | `4C 5A 34 42` (`LZ4B`) |
+| Multi-block sentinel | `4C 5A 34 4D` (`LZ4M`) |
+| Dictionary sentinel | `4C 5A 34 44` (`LZ4D`) |
+| LZ4M block size | 1 GiB (`0x40000000`) |
+| Max dictionary size | 8 KiB |
+| Min compress, no dict | 512 B |
+| Min compress, with dict | 128 B |
+| LZ4B envelope | 12 B (4 sentinel + 8 size) |
+| Uncompressed envelope | 4 B (sentinel only) |
 
 ## Performance
 
@@ -123,7 +168,7 @@ Lorem ipsum prefix) input.
 |   16 KiB   |       ~3.2 µs  |     ~2.4 µs |       ~3.9 µs  |     ~3.0 µs |
 |    1 MiB   |      ~89 µs    |    ~87 µs   |     ~173 µs    |   ~303 µs   |
 
-**End-to-end PUSH → PULL over `lz4+tcp://` (loopback):**
+**End-to-end PUSH -> PULL over `lz4+tcp://` (loopback):**
 
 | Message size | Throughput |
 |--------------|-----------:|
@@ -141,8 +186,8 @@ OMQ_DEV=1 bundle exec ruby --yjit bench/head_to_head.rb   # lz4 vs zstd
 
 ### Head-to-head vs `omq-zstd` and plain `tcp`
 
-End-to-end PUSH → PULL throughput, Ruby 4.0 + YJIT. Input:
-UUID-sprinkled Lorem ipsum — a fresh UUID between each Lorem
+End-to-end PUSH -> PULL throughput, Ruby 4.0 + YJIT. Input:
+UUID-sprinkled Lorem ipsum, a fresh UUID between each Lorem
 paragraph. Approximates realistic workloads where a schema
 repeats but values vary (event logs, protobuf records, JSON
 events), so a fraction of every message is mandatorily
@@ -160,27 +205,27 @@ across three bandwidth regimes.
 | Link                | Metric   |   tcp | lz4+tcp | zstd -3 | zstd 3 |
 |---------------------|----------|------:|--------:|--------:|-------:|
 | **100 Mbit**        | plain    |  11.8 |     105 |     114 | **197**|
-| (cap ≈ 12 MiB/s)    | wire     |  11.8 |      12 |      12 |    12  |
-|                     | speedup  | 1.00× |   8.89× |   9.70× |**16.74×**|
+| (cap ~12 MiB/s)     | wire     |  11.8 |      12 |      12 |    12  |
+|                     | speedup  | 1.00x |   8.89x |   9.70x |**16.74x**|
 | **1 Gbit**          | plain    | 117   |     794 | **900** |    603 |
-| (cap ≈ 125 MiB/s)   | wire     | 117   |      93 |      94 |     36 |
-|                     | speedup  | 1.00× |   6.81× |**7.73×**|  5.17× |
+| (cap ~125 MiB/s)    | wire     | 117   |      93 |      94 |     36 |
+|                     | speedup  | 1.00x |   6.81x |**7.73x**|  5.17x |
 | **Unlimited loopback** | plain | **1 064** |  869 |    972  |    626 |
 | (kernel-copy-bound) | wire     | 1 064 |      99 |     101 |     37 |
-|                     | speedup  | 1.00× |   0.82× |   0.91× |  0.59× |
+|                     | speedup  | 1.00x |   0.82x |   0.91x |  0.59x |
 
 Three regimes visible:
 
-- **100 Mbit** — all compressed transports saturate wire at
-  ~12 MiB/s. Plaintext = wire-cap × (1 / compression-ratio). The
+- **100 Mbit**: all compressed transports saturate wire at
+  ~12 MiB/s. Plaintext = wire-cap x (1 / compression-ratio). The
   tighter the ratio, the bigger the win: `zstd 3`'s 3% wire ratio
-  translates to a **~17× throughput multiplier** over plain tcp.
-- **1 Gbit** — compressed transports shift from wire-saturated to
-  CPU-limited. `zstd -3` reaches ~75% of wire cap; `zstd 3` only
+  translates to a **~17x throughput multiplier** over plain tcp.
+- **1 Gbit**: compressed transports shift from wire-saturated to
+  CPU-limited. `zstd -3` reaches ~75% of wire cap. `zstd 3` only
   29% (deep CPU-bound). Both beat plain tcp (which is pinned at
-  the wire cap) by **6–8×**. `zstd 3`'s tighter wire no longer
-  helps — there's no wire saturation to trade CPU for.
-- **Unlimited loopback** — no wire cap. All three are
+  the wire cap) by **6-8x**. `zstd 3`'s tighter wire no longer
+  helps. There is no wire saturation to trade CPU for.
+- **Unlimited loopback**: no wire cap. All three are
   CPU-limited. Plain tcp doesn't pay compression CPU, so **skip
   compression on loopback**.
 
@@ -196,25 +241,26 @@ Or use a `veth` pair in a network namespace so shaping doesn't
 touch your host's real loopback (see `tc-netem(8)`, `ip-netns(8)`).
 
 Full sweeps (8 sizes from 256 B to 512 KiB) for each regime live
-in `bench/head_to_head.rb` output — run it yourself; the
+in `bench/head_to_head.rb` output. Run it yourself. The
 headline numbers above are stable across repeats but small sizes
 and very large sizes vary a bit run-to-run.
 
 **Takeaway:**
 
 - Pick **`lz4+tcp://`** for bandwidth-limited links (any real
-  network — even 1 Gbit LAN). 6–9× throughput multiplier over
+  network, even 1 Gbit LAN). 6-9x throughput multiplier over
   plain `tcp`, minimal memory (~16 KiB/connection), modest CPU.
-  Ties or beats `zstd -3` at 1 Gbit; loses the ratio race to
+  Ties or beats `zstd -3` at 1 Gbit. Loses the ratio race to
   `zstd 3` at 100 Mbit and below.
-- Pick **`zstd+tcp://` (level ≥ 3)** when the wire is the
-  precious resource (≤ 100 Mbit links, WAN, or you're paying for
-  egress). **~17× throughput multiplier at 100 Mbit** for 128 KiB
-  messages is hard to argue with.
+- Pick **`zstd+tcp://` (level >= 3)** when the wire is the
+  precious resource (100 Mbit links or slower, WAN, or you're
+  paying for egress). **~17x throughput multiplier at 100 Mbit**
+  for 128 KiB messages is hard to argue with. Treat this as
+  experimental in Ruby OMQ.
 - Pick **plain `tcp://`** when the link is *not* the bottleneck
   (localhost IPC, loopback, datacenter-fast inter-host
   connections where the bandwidth ceiling is above the CPU's
-  compress/decompress speed — typically 10+ Gbit), or when the
+  compress/decompress speed, typically 10+ Gbit), or when the
   payload is already high-entropy (encrypted, already compressed,
   random binary) and compression only adds overhead.
 

data/lib/omq/lz4/codec.rb

@@ -1,6 +1,6 @@
 # frozen_string_literal: true
 
-require "rlz4"
+require "lz4rip"
 
 require_relative "errors"
 
@@ -41,7 +41,7 @@ module OMQ
       # passthrough anyway. Below the threshold, `encode_part` emits
       # UNCOMPRESSED directly without touching the compressor.
       MIN_COMPRESS_NO_DICT   = 512
-      MIN_COMPRESS_WITH_DICT = 32
+      MIN_COMPRESS_WITH_DICT = 128
 
       # Maximum dictionary size on the wire. A policy choice, not a
       # protocol limit; tight enough that constrained peers can accept
@@ -62,7 +62,7 @@ module OMQ
       # back to passthrough when compression wouldn't save at least the
       # envelope overhead.
       #
-      # `block_codec` is an RLZ4::BlockCodec, optionally constructed with
+      # `block_codec` is an Lz4rip::BlockCodec, optionally constructed with
       # `dict: bytes`. The codec's dict presence is detected via
       # `#has_dict?` to pick the min-size threshold.
       #
@@ -120,7 +120,7 @@ module OMQ
           block = wire_bytes.byteslice(12, wire_bytes.bytesize - 12)
           begin
             block_codec.decompress(block, decompressed_size: decompressed_size)
-          rescue RLZ4::DecompressError => e
+          rescue Lz4rip::DecompressError => e
             raise ProtocolError, "LZ4B decode failed: #{e.message}"
           end
         when LZ4M_SENTINEL
@@ -213,7 +213,7 @@ module OMQ
             block_decompressed_size = [block_size, remaining].min
             begin
               output << block_codec.decompress(block_data, decompressed_size: block_decompressed_size)
-            rescue RLZ4::DecompressError => e
+            rescue Lz4rip::DecompressError => e
               raise ProtocolError, "LZ4M block decode failed: #{e.message}"
             end
 

data/lib/omq/lz4/version.rb

@@ -2,6 +2,6 @@
 
 module OMQ
   module LZ4
-    VERSION = "0.3.0"
+    VERSION = "0.3.2"
   end
 end

data/lib/omq/transport/lz4_tcp/connection.rb

@@ -12,30 +12,44 @@ module OMQ
       # receive_message route through compression, but everything else
       # (write_command, close, etc.) passes through untouched.
       class Lz4Connection < SimpleDelegator
+        DEFAULT_DICT_CAPACITY = 2048
+        DEFAULT_TRAIN_TRIGGER = 100
+
         # @return [Integer, nil] wire bytesize of the last received
         #   message (sum across parts, compressed sentinel included).
         attr_reader :last_wire_size_in
 
-        def initialize(conn, send_dict_bytes:, max_message_size:)
+        def initialize(conn, send_dict_bytes:, max_message_size:, auto_dict: nil)
           super(conn)
-          @max_message_size  = max_message_size
-
-          # Per-direction state. The send codec is built at construction
-          # with the send-side dictionary (if any) baked in. The receive
-          # codec starts dict-less; if/when a dict shipment arrives on
-          # this direction, it is replaced with a dict-bound codec.
-          @send_dict_bytes   = send_dict_bytes&.b
-          @send_codec        = build_block_codec(@send_dict_bytes)
-          @send_dict_shipped = @send_dict_bytes.nil?  # nothing to ship => "already shipped"
-
-          @recv_codec        = build_block_codec(nil)
-          @recv_dict_bytes   = nil
+          @max_message_size = max_message_size
 
+          @recv_codec       = build_block_codec(nil)
+          @recv_dict_bytes  = nil
           @last_wire_size_in = nil
+
+          if send_dict_bytes
+            @send_dict_bytes   = send_dict_bytes.b
+            @send_codec        = build_block_codec(@send_dict_bytes)
+            @send_dict_shipped = false
+            @trainer           = nil
+          elsif auto_dict
+            @send_dict_bytes   = nil
+            @send_codec        = build_block_codec(nil)
+            @send_dict_shipped = true
+            @trainer           = Lz4rip::DictTrainer.new(auto_dict[:capacity])
+            @train_trigger     = auto_dict[:trigger]
+            @train_msg_count   = 0
+          else
+            @send_dict_bytes   = nil
+            @send_codec        = build_block_codec(nil)
+            @send_dict_shipped = true
+            @trainer           = nil
+          end
         end
 
 
         def send_message(parts)
+          maybe_train!(parts)
           wire = encode_parts(parts)
           ship_send_dict!
           __getobj__.send_message(wire)
@@ -43,6 +57,7 @@ module OMQ
 
 
         def write_message(parts)
+          maybe_train!(parts)
           wire = encode_parts(parts)
           ship_send_dict!
           __getobj__.write_message(wire)
@@ -50,6 +65,7 @@ module OMQ
 
 
         def write_messages(messages)
+          messages.each { |parts| maybe_train!(parts) }
           wires = messages.map { |parts| encode_parts(parts) }
           ship_send_dict!
           __getobj__.write_messages(wires)
@@ -76,10 +92,36 @@ module OMQ
 
         def build_block_codec(dict_bytes)
           if dict_bytes
-            RLZ4::BlockCodec.new(dict: dict_bytes)
+            Lz4rip::BlockCodec.new(dict: dict_bytes)
           else
-            RLZ4::BlockCodec.new
+            Lz4rip::BlockCodec.new
+          end
+        end
+
+
+        def maybe_train!(parts)
+          return unless @trainer
+
+          parts.each do |pt|
+            bytes = pt.is_a?(String) && pt.encoding == Encoding::BINARY ? pt : pt.to_s.b
+            @trainer.add_sample(bytes)
           end
+          @train_msg_count += 1
+          return if @train_msg_count < @train_trigger
+
+          finish_training!
+        end
+
+
+        def finish_training!
+          dict_bytes = @trainer.train
+          @trainer = nil
+
+          return if dict_bytes.empty?
+
+          @send_dict_bytes   = dict_bytes.b.freeze
+          @send_codec        = build_block_codec(@send_dict_bytes)
+          @send_dict_shipped = false
         end
 
 
@@ -137,9 +179,9 @@ module OMQ
 
           dict_bytes = LZ4::Codec.decode_dict_shipment(wire)
           # Replace the no-dict recv codec with a dict-bound one. The
-          # old codec is GC'd. Per rlz4: "fresh codec when dict is
-          # needed" — BlockCodec's dict is a permanent property.
-          @recv_codec      = RLZ4::BlockCodec.new(dict: dict_bytes)
+          # old codec is GC'd. Lz4rip treats dict as a permanent codec
+          # property, so install-time is always a fresh build.
+          @recv_codec      = Lz4rip::BlockCodec.new(dict: dict_bytes)
           @recv_dict_bytes = dict_bytes
         end
 

data/lib/omq/transport/lz4_tcp/transport.rb

@@ -29,9 +29,15 @@ module OMQ
         #   ship on the first outgoing message. If nil, no dict is
         #   shipped (payloads compress without dict or go plaintext
         #   below the min-size threshold).
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training from early traffic (RFC §7.6). Pass `true` for
+        #   defaults (capacity=2048, trigger=100) or a Hash with
+        #   `:capacity` and/or `:trigger` keys. Mutually exclusive
+        #   with `dict:`.
         # @return [Listener]
-        def listener(endpoint, engine, dict: nil, **)
+        def listener(endpoint, engine, dict: nil, auto_dict: nil, **)
           validate_dict!(dict)
+          validate_auto_dict!(auto_dict, dict)
 
           host, port  = parse_endpoint(endpoint)
           lookup_host = normalize_bind_host(host)
@@ -46,7 +52,7 @@ module OMQ
           host_part    = display_host.include?(":") ? "[#{display_host}]" : display_host
           resolved     = "#{SCHEME}://#{host_part}:#{actual_port}"
 
-          Listener.new(resolved, servers, actual_port, engine, dict&.b)
+          Listener.new(resolved, servers, actual_port, engine, dict&.b, normalize_auto_dict(auto_dict))
         end
 
 
@@ -55,10 +61,13 @@ module OMQ
         # @param endpoint [String]
         # @param engine [OMQ::Engine]
         # @param dict [String, nil] user-supplied dictionary bytes.
+        # @param auto_dict [true, Hash, nil] enable automatic dictionary
+        #   training. See `listener` for details.
         # @return [Dialer]
-        def dialer(endpoint, engine, dict: nil, **)
+        def dialer(endpoint, engine, dict: nil, auto_dict: nil, **)
           validate_dict!(dict)
-          Dialer.new(endpoint, engine, dict&.b)
+          validate_auto_dict!(auto_dict, dict)
+          Dialer.new(endpoint, engine, dict&.b, normalize_auto_dict(auto_dict))
         end
 
 
@@ -111,6 +120,40 @@ module OMQ
           raise LZ4::ProtocolError,
             "dict size #{size} out of range [1, #{LZ4::Codec::MAX_DICT_SIZE}]"
         end
+
+
+        def validate_auto_dict!(auto_dict, dict)
+          return unless auto_dict
+
+          if dict
+            raise ArgumentError, "cannot combine auto_dict: and dict:"
+          end
+
+          return if auto_dict == true
+
+          unless auto_dict.is_a?(Hash)
+            raise TypeError, "auto_dict: must be true or a Hash; got #{auto_dict.class}"
+          end
+
+          cap = auto_dict[:capacity]
+          if cap && (cap < 1 || cap > LZ4::Codec::MAX_DICT_SIZE)
+            raise ArgumentError,
+              "auto_dict capacity #{cap} out of range [1, #{LZ4::Codec::MAX_DICT_SIZE}]"
+          end
+        end
+
+
+        def normalize_auto_dict(auto_dict)
+          return unless auto_dict
+
+          capacity = Lz4Connection::DEFAULT_DICT_CAPACITY
+          trigger  = Lz4Connection::DEFAULT_TRAIN_TRIGGER
+          if auto_dict.is_a?(Hash)
+            capacity = auto_dict[:capacity] if auto_dict[:capacity]
+            trigger  = auto_dict[:trigger]  if auto_dict[:trigger]
+          end
+          { capacity: capacity, trigger: trigger }.freeze
+        end
       end
 
 
@@ -118,10 +161,11 @@ module OMQ
       class Dialer
         attr_reader :endpoint
 
-        def initialize(endpoint, engine, dict_bytes)
-          @endpoint   = endpoint
-          @engine     = engine
+        def initialize(endpoint, engine, dict_bytes, auto_dict)
+          @endpoint  = endpoint
+          @engine    = engine
           @dict_bytes = dict_bytes
+          @auto_dict = auto_dict
         end
 
 
@@ -144,6 +188,7 @@ module OMQ
             conn,
             send_dict_bytes:  @dict_bytes,
             max_message_size: @engine.options.max_message_size,
+            auto_dict:        @auto_dict,
           )
         end
       end
@@ -153,13 +198,14 @@ module OMQ
       class Listener
         attr_reader :endpoint, :port
 
-        def initialize(endpoint, servers, port, engine, dict_bytes)
-          @endpoint   = endpoint
-          @servers    = servers
-          @port       = port
-          @engine     = engine
+        def initialize(endpoint, servers, port, engine, dict_bytes, auto_dict)
+          @endpoint  = endpoint
+          @servers   = servers
+          @port      = port
+          @engine    = engine
           @dict_bytes = dict_bytes
-          @tasks      = []
+          @auto_dict = auto_dict
+          @tasks     = []
         end
 
 
@@ -168,6 +214,7 @@ module OMQ
             conn,
             send_dict_bytes:  @dict_bytes,
             max_message_size: @engine.options.max_message_size,
+            auto_dict:        @auto_dict,
           )
         end
 

data/lib/omq/transport/lz4_tcp.rb

@@ -11,7 +11,7 @@
 #   push.connect("lz4+tcp://127.0.0.1:5555")
 
 require "omq"
-require "rlz4"
+require "lz4rip"
 
 require_relative "../lz4/errors"
 require_relative "../lz4/codec"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: omq-lz4
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.3.2
 platform: ruby
 authors:
 - Patrik Wenger
@@ -15,31 +15,31 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.23'
+        version: '0.28'
 - !ruby/object:Gem::Dependency
-  name: rlz4
+  name: lz4rip
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.5'
+        version: 0.1.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.5'
-description: 'Adds lz4+tcp:// endpoint support to OMQ with per-part LZ4 block-format
-  compression, bounded decompression, and in-band dictionary shipping. Complementary
-  to omq-zstd: worse ratio, far faster encode, far smaller per-connection footprint.'
+        version: 0.1.1
+description: Adds lz4+tcp:// endpoint support to OMQ with per-part LZ4 block-format
+  compression, bounded decompression, in-band dictionary shipping, and automatic dictionary
+  training.
 email:
 - paddor@gmail.com
 executables: []
@@ -56,7 +56,7 @@ files:
 - lib/omq/transport/lz4_tcp.rb
 - lib/omq/transport/lz4_tcp/connection.rb
 - lib/omq/transport/lz4_tcp/transport.rb
-homepage: https://github.com/paddor/omq-lz4
+homepage: https://github.com/zeromq/omq.rb/tree/main/gems/omq-lz4
 licenses:
 - ISC
 metadata: {}
@@ -67,14 +67,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.3'
+      version: '4.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.6
+rubygems_version: 4.0.16
 specification_version: 4
 summary: LZ4+TCP transport for OMQ
 test_files: []